opendata

Open Data: Let’s talk about more than just government

Open Data: Let’s talk about more than just government

Byoliveridley

Victoria is hosting its open data day and Hackathon Saturday the 23rd (Facebook Link). I plan on being there because I support openness and transparency, I’d like to learn more about available data sets, and hangout with like-minded people. The City of Victoria has taken steps since 2011 through Councillor Marianne Alto‘s initiatives and more to facilitate more open governance. Like any other government entity, there is valid criticism and issues to navigate, but stated goals exist and progress can be tracked and critiqued.

Enough people talk about open government data, and there’s consensus that governments should be more collaborative, open and participatory. But most of us spend more time and money interacting with non-government entities than we do with government entities. Look at your monthly budget. You will spend 30-40 percent on your mortgage or rent, goes to a non-government entity. The next biggest line items, probably groceries, car payments are all to private entities. Should we as consumers not expect the same open data sharing standards from our private entities as we do from government? The book Open Government, released for free by Safari books after Aaron Swartz’s death (does not appear to be free any more) has one chapter by Archon Fung and David Weil titled Open government and Open Society, which outlined my concerns very well:

Enthusiasts of transparency, which most readers of this book are, should be aware of two major pitfalls that may mar this achievement. The first is that government transparency, though driven by progressive impulses, may draw excessive attention to government’s mistakes and so have the consequence of reinforcing a conservative image of government as incompetent and corrupt. The second is that all this energy devoted to making open government comes at the expense of leaving the operations of large private sector organizations—banks, manufacturers, health providers, food producers, drug companies, and the like—opaque and secret. In the major industrialized democracies (but not in many developing countries or in authoritarian regimes), these private sector organizations threaten the health and well-being of citizens at least as much as government.

Open Government – Chapter 8 – Open Government and Open Society – Fung and Weil

I wrote briefly about one aspect of open data in our private interactions, shopping receipts. We spend a lot of time, effort and money shopping, yet we’re very unlikely to leverage the power of data to help us shop better because our individual decisions are captured in paper receipts. But there are many more examples.

  1. Mortgages – Do you have to go to every bank/lender’s website to do a comparison? Ratehub is a start, is there an API or download capabilities?
  2. Real Estate Data – Realtors control real estate data in Canada, I would call this a major conflict of interest. There are efforts to open this data up a la the US, but slow going. This is the biggest market transaction any of us will undertake in our lives, but information is controlled by the agency that benefits most from our lack of knowledge.
  3. Rentals – Craigslist is notorious for hoarding data and going after people who want to present data in more useful formats. Community posted information is created by the community, but captured by private entities due to network effects (everyone’s on craiglist, so I need to be there too, regardless of their data policies).
  4. Insurance markets – Government provided insurance information (ICBC – Car, MSP – health) is transparent. Try getting insurance in the open market for condos, homes and more, you’ll find the same pdf/paper quote formats that make it difficult to compare and choose wisely.
  5. Corporate governance – There is so much information missing on actual corporate structures, ownership, directorship, brand ownership and lobbying
  6. Pollution and resource use. Do we have a good idea what companies pay for water or power? Do we have a way of understanding who pollutes what and where?

My goal on open data is to advocate for openness in all of society, not just in government. Also, just because data is available does not mean it is open. APIs and download capabilities are key.

So, when you think open data, do try and shift your gaze away from government occasionally. Remember that your housing decision is much more critical than the salary information for the assistant city manager, so openness is vital everywhere.

cart

 

Update: as Kevin pointed out on twitter, the federal tax bill is pretty big. I was talking more in terms of the municipal parts like property taxes. The point nevertheless stands, we pay private entities large sums of money under poor data transparency conditions.

Facebook and deep customer tracking, I want my data!

Facebook and deep customer tracking, I want my data!

Byoliveridley

To nobody’s surprise, Facebook, just like any other entity selling you stuff, or selling you to people who sell you stuff is trying to connect more and more sets of previously unconnected data. This particular case deals with brick and mortar store data that is linked with customers’ email addresses and loyalty cards.

Facebook will be using Datalogix to prepare reports for its advertisers about who, if anyone, bought more of their stuff after they ran ads on the social network. But by matching your Facebook profile with your CVS bill, this means that Facebook has the potential to know some of your most intimate details (my, that’s a lot of bunion cream you’re buying!), and the privacy concerns are enormous. When DoubleClick attempted something similar to this, user-backlash ultimately led them to cancel the project.

Can Facebook Possibly Build a Business Model That Isnt Inherently Creepy? – Derek Thompson – The Atlantic.

Corporations (more than government, open data activists!) have been deep mining our data for years. It is part of creating the information asymmetry that enables profits to be made. You may remember this story about Target (coming to Canada as soon as we can learn to say Tarjay) and how they outed a teen’s pregnancy.

 About a year after Pole created his pregnancy-prediction model, a man walked into a Target outside Minneapolis and demanded to see the manager. He was clutching coupons that had been sent to his daughter, and he was angry, according to an employee who participated in the conversation.

“My daughter got this in the mail!” he said. “She’s still in high school, and you’re sending her coupons for baby clothes and cribs? Are you trying to encourage her to get pregnant?”

Companies’ ability to reduce us to a shopping probability statistic is only going to get better as they learn to connect more of our data and computing gets faster. Can regulation keep up? Can customer outrage keep up with companies offering us coupons to keep us temporarily happy as impulse centres in our brain are carefully triggered for profit? Can customer outrage even keep up with the barrage of occurrences?

Receipt 2.0

I don’t think we can keep up. So, I want my data. I want information on what I bought, when I bought it, where I bought it in a standardized open data format. No, not a paper receipt, not a paper receipt scanner, but something that can be beamed to my phone, or emailed to me. I want to know when I buy coffee. Can I correlate my shopping habits with my mood?  Do I buy more random electronics when I need a pick me up? What is the spread in the price I paid for my favourite cereal? Do some stores price it differently on Wednesdays? I want apps that can mine my data and tell me where to buy my cereal, or when not to buy. I want apps that can tap into a product database and give me a carbon footprint, or a fair trade pass/fail, or a local product breakdown.

Also, I do not want to re-enter the same bits of data multiple times and increase error. A payment made to my dentist should be sent directly to my extended health “insurance” provider for a refund. It should also go to my tax receipts virtual pile and await reimbursement. Any tax deductions can easily be tagged and directly entered into my tax preparation software at the end of the year. if I want to expense something for work, I should just be able to tag them and send them along. I can’t imagine how much time and effort this will save in error checking, manual entry, auditing, so much more.

Some of this is possible with systems like Mint, but they operate on a payment level, not on a line item level.

The new mobile payment system Square (not in Canada yet) shows some potential, so does Intuit’s GoPayment, which is available in Canada. But these payment systems emphasize ease of payment on both sides of the transaction, not the ability to mine our own data.

Can this happen via the market with no regulatory push? I don’t see how. Reducing information asymmetry is not in corporations’ interest. So it will have to be regulated. You have our data, just give it to us.

Not to mention, this is the “free market” way to go. Think of all the innovation that can be unleashed on the consumer side. Think of the apps that can provide better financial advice, the apps that can collate data at city/regional level and help consumers make better decisions.

Will companies have to spend money to make this happen? Yes. This will not be challenging for larger companies who already spend millions deep-mining our data. What about small business? This is where small tweaks to new systems like Square or GoPayment can be the game changer. Square already charges less for a swipe than a typical Visa transaction. So, I would see local business as saving money. When I buy local, I usually feel a bit more connection with the product. Imagine seeing my decision reinforced by data from Receipt 2.0. Small local business cannot data mine, but can generate enough goodwill with local consumers to get access to their data. Unlike Target, when my corner soap store does something unethical with my data, they cannot survive the bad publicity.

Are there privacy concerns? Yes. But our privacy is already compromised the minute we pay anything other than cash, or use the internet (Tor users, stop smirking). Good regulation can address most concerns.

My information is being used to make me a consume more, I want it to make me a better “consumer”. Open data efforts have focused intensely on public and government data, while privacy activists have tried unsuccessfully to stop private data gathering. I would like open data advocates to look carefully at liberating corporate mined data as well.

Featured Image courtesy the Culturally Authentic Picture Lexicon used under a creative commons licence.