The city will officially announce today that the City Record, which has published procurement, hearing and other public notices weekdays since 1873, now has a fully searchable, expanded and more accessible online presence.

The new platform is the result of legislation signed into law last year that required the publication, produced by the Department of Citywide Administrative Services, to be available online in a searchable and machine-readable format, and which was introduced by Councilman Ben Kallos.

...

The expanded database will include public hearings and meetings on subjects such as zoning changes, liquor licenses and sidewalk cafe applications, regulatory changes, hiring notices and other city personnel changes. The platform also allows users to download attached documents such as agendas and requests for proposals when necessary. The print edition will continue to be available. Users will be able to sign-up to receive e-mail notifications for areas of interest, and the information will also be available through the city's open data portal.

...

Kallos praised an "unprecedented level of cooperation" on the project between the Council, the administration, as well as the developer community. He drew on his own background as a software developer to assist with work on developing schemas, normalizing the publication's data input, and engaging with DCAS to go over their front-end and back-end designs and recommend additional features.

Beyond the first step of simply making the data available online in a machine-readable format, he worked with DCAS on adding geographic information. "A lot of the information has previously been just in one text block," he noted. "[With the] new launch, the location of the hearings will actually be parsed out in address format...I'm hoping that in our next release, the geographic information relating to the items being heard will be the next frontier."

Improved access to the City Record will help provide notice "so that people in communities can give their feedback [on neighborhood proposals] at the right time before it's too late," Kallos said. "The City Record is the most important newspaper that no one has ever heard of and hopefully it will become the dataset that everyone uses even if they don't even know it because they're using it through an app that helps make the information more useful," he added, saying that he would be listening for feedback on future improvements.

The civic technologists' involvement began when the Mayor's Office reached out to BetaNYC about converting older PDF versions of the City Record going back to around 1998 into machine-readable data, Noel Hidalgo executive director of BetaNYC, recalled.

"As we got access to the PDFs, we realized that we actually have a much larger project ahead of us, it's not just being able to scrape PDFs and turn them into some magical data," he said. As the group worked on developing tools that would be able to extract information both from data already digitized and the older PDFs, a more complex process, Hidalgo said one challenge was the lack of an overall glossary of all the different kinds of information in the City Record. The process of extracting data in a structured from the publication by is like "teaching a robot how to walk," he said.

In over 20 meetings with DCAS, developers highlighted the functionality of an existing civic notification tool called Citygram to demonstrate the importance of adding date and address fields to the meeting notices.

"This was the first time that the city invited technologists to come in and work intimately with them," he said. "Even though we may have not had all the resources on either side to fully execute the intended vision [in the timeframe of a year], we were able to accomplish a significant amount of work," he said, pointing to creation of several tools, such as a parser that the developers can start using to create online hearing notices. "That's a huge, huge win," Hidalgo said. "Now that the pressure of the anniversary of this law has passed, we are looking very much forward to the city making a greater commitment to ... making the City Record more accessible."