Now… there are some little things you should know before writing JSON parser:

Take a look at these:

char
any-Unicode-character-
except-”-or-\-or-
control-character
\”
\\
\/
\b
\f
\n
\r
\t
\u four-hex-digits

There’s one escaped char that doesn’t conform to the reality and there’s another char that should be escaped. Could you tell me what are these chars?

Hint: there are actually two chars that should be escaped. Answers are below:

One: forward slash shouldn’t be escaped. Moreover, JSON validators like jslint.org and jsonlint.org accept unescaped forward slashes just fine. The reason for this requirement is that escaped forward slashes add complexity to parsing and reading URLs embedded in JSON.

Two: angle brackets < and > should be escaped. Reason: security. For example, IIS (of some versions) will refuse to accept incoming requests with angle brackets inside. Escape or die.

Oh, and you can escape these brackets in any way, like &gt or \u0xxx , but parser (at least parser trying to be true to the spec) will convert only \u0xxx form.

PS Programming JSON writer API in COM is an exercise in humility. WriteLong(), WriteDouble, WriteString,WriteLongVal, WriteDoubleVal, WriteStringVal…

- real programming is C and C++ on tasks where every mistake could be fatal (graphic drivers?)
- .Net and Java are scripting languages.

O-ok…

No, I am not proving any point here, I just think that labeling does not solve anything.

First, I want to quickly summarize this article: JSON vs XML: confessions of a JSON advocate.

1. Absence of comments. I really can’t comment on this.
2. Absence of attributes. In short, there’s always some properties you want to read first. XML ensured that these properties are available by putting them in attributes. No such luck in JSON.
3. Unquoted linefeeds are not allowed in JSON strings.
4. No ordering except of arrays. See my comment to #2, but that one is not about attributes byt rather xs:sequence.

As I told you earlier, most web service writers and consumers don’t even notice these limitations. If you send out stock price ticker in JSON, there’s no need in comments, attributes or sequential containers, you just parse entire thing into a DOM-like structure and use it.

However, there are people who are forced to overcome such problems. In this post, I’ll try to show a best JSON data container structure that can help most of the people writing JSON-based protocols for the web data exchange.

Let’s start with an example of the most common issue – large data and important metadata. My example is a spatial data container. It has a seemingly problem-free syntax: http://help.arcgis.com/en/arcgisserver/10.0/apis/rest/fsquery.html

This REST operation returns JSON featureset which is a simple JSON object. Bulk of the data in this object is a ‘features’ array that contains (obviously) features. All properties before ‘features’ is metadata. These properties are needed to parse the ‘features’ array.The ‘features’ array usually takes 99.9% of the total size of a container.

Now.. what if your response is big? Or even huge?

Most obvious solution is to amend the protocol rules and introduce a limit on a number of features. But there are cases where feature limit cannot be used. For the sake of an example let’s suppose that a pure, innocent soul decided to use this operation to replicate entire database and turned featurelimit off. Or worse, this protocol has no continuation clause, which means that search-engine-style pagination is not available out-of-the-box. (Yes there are tricks to emulate pagination by requesting all feature IDs first… but I am not talking about these right now.)

While it is possible to read entire featureset into memory, it makes much more sense to use a sequential reader. SAX, not DOM.

Now that I said the word SAX, I should apologize. I didn’t really mean SAX, I actually mean simple sequential one-pass reader, like Microsot .Net’s XmlReader. The sequential reading has simple advantage of not keeping entire DOM-like data tree in memory. But it requires a sequential data container. Oh xs:sequence, where art thou when we need thou? Microsoft had to get around this even for XML – they introduced XmlBookmarkReader. Basically, it allows you to reset the reader state to any arbitrary position. It’s not cost-free, because every time you go back, you wasted a pass. While skipping tokens in a string is less costly in terms of memory (vs DOM model), it still takes CPU cycles to perform.

Can we avoid double-pass parsing? Can we use JSON syntax to create a container that provides sequential access to the data?

Yes, JSON actually provides an anonymous sequential container: JSON array.

Updated data container syntax will look like this:

[

 {

 "importantProperty" : "<importantPropertyValue>"

 },

 [ //anonymous array that used to be ‘features’.

  <feature1>, <feature2>

 ]

]

See? We enforced metadata to come before data without violating JSON syntax rules.

Now there’s a bigger problem: types

If you are not a JSON novice like me, you probably noticed that most of serializers that work with JSON introduce a special __type property that holds a type hint for the deserializer. (Examples: OpenLayers, Microsoft WPF JSON deserializer). Apparently, duck typing is not always an option when you want to (painlessly) introduce automatic serialization and code generation. And versioning support for a good measure.

But where do you insert __type in arrays? It only works for objects.

A naive solution would be wrap a sequential container in an object with __type property. However, __type property could very well come last. We just returned to the square one.

When in doubt, look at the giants. How Microsoft got around this? By introducing a DataContract attribute language which is essentially a generalized XML-like schema. In essence, you provide a data contract and expect the server to be conforming to that contract. Typical workflow is to make a first request to the web service by asking a proper version, then use proper DataContract-attributed value type (class) to deserialize a service response.
Aha! This is DOM-like model again! Looks like there’s no way around it…

Unless we introduce type&version string as a very first mandatory element of the array (use null for the default (lowest) version).

Sequential data container, version 3:

[

 “sequential.data.container.3”, //type and version

 {

  "importantProperty" : "<importantPropertyValue>"

 },

 [ //anonymous array that holds the rest of the data.
  //Use importantProperty and type information to read it properly

  <feature1>, <feature2>

 ]

]

Yes, this looks ugly, but this is the best we can do without violating JSON syntax.
Now let’s remember why we did all of this. We actually wanted a data structure that we can read and deserialize sequentially in one pass without introducing either intermediate storage (DOM-like model: read stream into an intermediate storage, then read intermediate storage in a specific order) or bookmarks (read stream once to read important parts, read stream once to read the rest).

Thankfully, most of the objects that people work with do not require such exotic treatment because their size is pretty much fixed.

Still, I am not sure if the end result was worth it. Basically, we just got rid of the property names and the result looks confusing. Well, if not confusing, then it’s certainly is not self-describing like XML and (most of the time) JSON are.

To actually solve this problem we need to look higher, beyond the (DOM-like) tree we just tried to climb and see the forest behind it. The answer is simple: JSON and XML are not designed for an efficient bulk data transfer but rather for an interoperability. If you want to transfer large amount of data, you should really use a binary protocol. Binary data transmission is well-researched area (think video compression and streaming) and you shouldn’t have any problems choosing the right protocol for you. We can look into this in the next article, if there is a need.

Exercise for the reader: Name couple of reasons why the sequential container designed in this article is bad.

Also, feel free to tell me where I was wrong. I am usually wrong, so there will be no hurt feelings.

A: Say… do you really need template programming in order to concatenate some strings? or, god forbid, you need to tokenize the std::wstring and have to write a method for that? Of you need a stringBuilder-like functionality?

B: C++0x standard is going to be old before it will be released. Come on guys, I am not even talking about fancy things like closures here. I am tired of not having rvalues. I am tired of not having static constructors. I am tired of non-human-readable errors of template-based metaprogramming. I am just tired of having pointers standing in my way 80% of time.

C: Do you know what takes most time for me when I work in C++? To replicate some functionality I take for granted in managed code like Java or .Net.

Example:

//Error.h

class Error
{
public:
 int code;
 Error(int c){code = c;};
 static const GenericError;
 static const DisplayError;
}

//Error.cpp

static map<int, const Error*> g_errMap;

class add_values_to_map
{
public:
add_values_to_map(map<int, const Error*>& _map, int& code, const Error& err)
  {
  map[code] = &err;
  }
};

static map<int, const Error*> g_errMap;
static const Error::GenericError(-1);
static const add_values_to_map tmp1(-1, Error::GenericError);

int main()
{
  Error* p = find_predefined_err_by_code(-1);
}

You probably noticed that this code replaces a static constructor without actually having one. Quite weird half-baked functor class… but this is the best i could come up with.

I’ll hug you if you give me a better solution.

NOTE1: No, I don’t want to call MyFrameworkInit(). Actually, I don’t have the global init method and I intend to keep it that way.
NOTE2: I don’t want to init static variables on the first call to class XX or YY or ZZ or in the non-static constructor of Error. Why? Because I don’t want to introduce any locks in my code.

Quite interesting strategy. Either it is promoting motorbike forum (which is unlikely) or it slowly infiltrates through the spam filters around the world. In couple of years account will reach maturity and the owner will deliver a message…

Or maybe someone just testing improved AI text generator on Markov chains.

WITH T1 as ( SELECT SNO AS SA, CITY FROM S ) ,
T2 as ( SELECT SNO AS SB, CITY FROM S ) ,
T3 as ( SELECT * from T1 NATURAL JOIN T2 ) ,
T4 AS ( SELECT * FROM T3 WHERE SA < SB )
SELECT SA, SB FROM T4

This is what you get from reading CISP and ‘SQL and Relational theory‘ in a row.

PS

Dreamed about newbie installing Linux on himself and attracting zillions of nuclear cyber-cooties. The moral was : always use Windows!

Palm Springs CC was most likely (forgive me for not checking Wikipedia) built during the shiny days of city’s glory, when all Hollywood hotshots loved to spend their weekends here. The style is unmistakable. Now about the favorite topic of every convention crasher – free food. Free food was in abundance. I was there only for one day (Wed Mar 24 2010), for that day ESRI provided free boxed lunch, drinks and snacks before and after lunch and free dinner (including beer and wine) on Dev Summit Attendee party. Beer was from Hangar 24, my favorite local brewery, and food was definitely of acceptable quality.

I attended several sessions presented by my colleagues, also had a good deal of talks in exhibit hall. (Exhibit hall is so small cozy that notion of ‘islands’ was close to non-existence, let’s just say I was there for significant chunk of my day).

Dev Summit is like boot camp for developers who want to get a glimpse of ArcGIS 10 early. Pre-release will be available soon and there’s plenty of other related activity happening – dev APIs are getting public as well.

My colleagues and friends Julio Andrade and Rex Hansen delivered a great presentation about extending ArcGIS Server .Net with custom Server Object Extensions.  I helped to write some of the related code (JSON/AMF reading/writing), so I attended as a volunteer cheerleader. I couldn’t find the session recording online, but I think it’s going to be available soon for attendees (if not for everyone.)

Julio did great job with helper objects in REST SOESupport assembly. As a result, SOAP and REST SOE samples were concise and clear. If you are curious, these samples will be available from the code gallery for 10. Again, no link yet. Check server blog from time to time.

Note to myself: raw access to IJSONWriter, IJSONReader is probably will be of some interest to people writing AGS SOEs, especially in C++. I should write separate posts about details of these new public interfaces.

One attendee asked me what SOE he should develop – SOAP or REST? Obviously REST, if you are creating web app for browser consumption. If you need SOAP, you can add it later to the same SOE.

Next session of note: Roop Goyal and Sud Menon presented “Editing Geodatabases over the Web Using ArcGIS Server“. Feature Service is pretty cool if you ask me. (Again, no recording available yet.).