Querying JSON using JSONPath

Keywords:
JSON query filter evaluate JSONPath expression

Problem:
Dealing with code that's doing a fair bit of JSON processing (traversing object structures and collating the values) it leads to the question - is there a standard way to lookup or filter data from the JSON structure using a query or expression?

The same question has been already been asked on StackOverflow: Is there a query language for JSON? and the answer is essentially that there are many approaches. There's another list of approaches with examples in 8 ways to query json structures.

So if there's no 'standard' approach (for example, backed by an RFC as is the case with JSON), which approach should I pick?


Solution:
JSONPath proposed by Stefan Goessner allows for XPath-like expressions to be evaluated against JSON. What's possible with the expression syntax seems to cover most of the traversal and collating scenarios I was interested in and the JavaScript implementation is remarkably lightweight (implemented as just one function jsonPath(<object>, <expression>, <optional arguments object>)).

Here some extra expression examples (beyond those shown in the JSONPath articles and jsonpath on Google Code):

Expression	Result
$	The root element (the same JSON back out - i.e. an identity transform).
$.*	All child elements of the root.
$.store.book	All book elements.
$.store.book[1]	The second book element.
$..price	The price of every element - result will be Numbers.
$.store.book[?(@.isbn=='0-553-21311-3')]	The book element where isbn is '0-553-21311-3'.
$.store.book[?(@.category=='fiction' && @.price > 10)]	The book element where category is 'fiction' and price is more than 10.

It seems like there's still some discussion around what's to be supported and there isn't exactly high activity on the project. So if using it, treat is as beta and evaluate if it does what you need. It's likely to be one of those things that once it gets past v0 there will be less (if any) changes to expect.

Some things to look out for:

• Where's the latest version?
  Although the downloads page says the latest version is 0.8.0, this few years old (2007). You can access later versions (of the JavaScript source) either via the SVN path to the trunk or via the Google Code Browser. The javascript implementation lists the version as 0.8.5 - I'm not sure what the cycle is for getting this to 'release'.
• No results returned as false
  I'd expect this to be simply an empty array [].
• Results as Arrays of Arrays
  If querying a nested JSON structure of objects and the matching results match different parts of the tree, what you get back is a nested array structure indicating the relative location of the matches. I'd prefer to get this back as a flattened array. I guess the logic is that have the context of where the items were found and if you want, you can post-process (and flatten) the Arrays of Arrays - some useful approaches discussed on StackOverflow:Merge/flatten an Array of Arrays in JavaScript?.
• the Object and it's properties in the results?
  In using the latest code mentioned above, it seems that - in some scenarios - where an object matches the expression the result includes the object as well as the properties as additional items in the result. For example:
  

  Instead of:	You get:
  [ { "category": "fiction", "author": "Herman Melville", "title": "Moby Dick", "isbn": "0-553-21311-3", "price": 8.99 } ]	[ { "category": "fiction", "author": "Herman Melville", "title": "Moby Dick", "isbn": "0-553-21311-3", "price": 8.99 }, "fiction", "Herman Melville", "Moby Dick", "0-553-21311-3", 8.99 ]

  This wasn't the case with 0.8.0 and may change in future releases?

Use javascript to set text (with newlines) into a textarea - for IE and Firefox

Keywords:
javascript jquery textarea DOM innerHTML newlines carriage return whitespace IE firefox chrome

Problem:
Setting text with newlines into a textarea using the innerHTML attribute:

<textarea id="source" rows="4" cols="50">
A
B
C
</textarea>
<button onclick="copyText();return false;">copy -></button>
<textarea id="target" rows="4" cols="50">
</textarea>


<script type="text/javascript">
    function copyText() {
        var sourceField = document.getElementById('source');
        var targetField = document.getElementById('target');
        targetField.innerHTML = sourceField.value;
    }
</script>

Works in "most" browsers but in IE the text put into the textarea has the newline characters stripped out (replaced with a single space character). Is there a way to make this work in IE? ... and most other browsers?

Solution:
To make it work in IE, setting the text via inputField.setAttribute("value", [your text]); will preserve the newlines (i.e. the "\n" character in javascript)

var sourceField = document.getElementById('source');
var targetField = document.getElementById('target');
targetField.setAttribute("value",sourceField.value);

Only problem is that setting the attribute alone is not enough for Firefox (& Chrome - all WebKit?). For the above example, the target field will not appear to have the value from the source (though the DOM will have the value). To get them all to work? Contrive the order in which you set things:

var sourceField = document.getElementById('source');
var targetField = document.getElementById('target');
var text = sourceField.value;
targetField.innerHTML = text; // now Firefox (& Chrome) are happy - but IE has the text as one line
targetField.setAttribute("value", text); // now IE has the text as multiple lines - the change should be imperceptible 

The same code using jQuery (note that jQuery's html() function will still use innerHTML ultimately so is subject to same IE issue with newlines being stripped):

jQuery(document).ready(function($){
    var text = $(sourceField).val();
    $(targetField).html(text);
    $(targetField).val(text);
}); 

Notes:
Use of innerHTML for managing textarea content - and keeping it in-synch with the DOM - was discussed in the previous post: Firefox does not reflect input form field values via innerHTML. I'll update that post accordingly ...

javascript object keys being sorted in some browsers

Keywords:
javascript object associative array map keys sorted ordered chrome safari webkit

Problem:
Is it the case that some browsers - WebKit, and possibly IE9 - are sorting the keys javascript objects (aka "associative-arrays", aka "maps")?

Trying the following code (note the integer keys as strings is intentional):

    var x = new Object;
    x["3"]="C";
    x["2"]="B";
    x["1"]="A";
    JSON.stringify(x);

Or as a 1-line snippet you can paste in a (modern) browser address bar:

    javascript:var x = new Object;x["3"]="C";x["2"]="B";x["1"]="A";JSON.stringify(x);

You'll get the following in Firefox:
{"3":"C","2":"B","1":"A"}

In Chrome you'll get:
{"1":"A","2":"B","3":"C"}

Is this a bug? Can you not expect the order the keys are added to be preserved?

Solution:
The short answer is no (though more appropriately but rude would be "why would you!?").

A translation in terms that a Java programmer may understand is think of the javascript object as a java.util.HashMap despite the fact it behaves a bit like a java.util.LinkedHashMap in Firefox and like a java.util.TreeMap in WebKit.

Don't code based on any expectation of the key order - which may need some thought if dealing with JSON representations of objects where there is an implicit order in the objects in "stringify-ied" form.

Some thoughts on how to maintain an order are here - How to keep an Javascript object/array ordered.

Javascript history.back() not working for iframes in Firefox

Keywords:
javascript history frame iframe go(-1) back firefox 3 top self

Problem:
You can get a hyperlink to behave as a "back" with a bit of javascript:

<a href="javascript:history.back()">back</a>

or (saying go back one place in the history - equivalent to the above):

<a href="javascript:history.go(-1)">back</a>

This works fine in all browsers when the page is in the main-view. If this javascript is encountered in an iframe you expect that you get the previous page that was loaded in that iframe. In firefox 3+ you're taken back in the main view page not the iframe - a nuisance if there's a detailed browsing history in the iframe view. Why does it work fine in other browsers and previous versions of firefox?

Solution:
I don't know why, but it seems that in firefox 3+ when you say history this becomes implicitly top.history. Someone has tried to raise this with mozilla with not much luck - javascript:history.back() and Firefox 3.0.1 and iframes.

The solution is to be explicit about which frame you want to navigate back with. If you just want all browsers to behave the same then self refers to the frame the user is clicking in (therefore in other browsers history is implicitly self.history). So the go-back hyperlink becomes:

<a href="javascript:self.history.back()">back</a>

or:

<a href="javascript:self.history.go(-1)">back</a>

This will behave the same regardless of if the page is loaded in the main-view or in an frame/iframe.

GWT Button acts as a submit in WebKit browsers

Keywords:
HTML button element form WebKit Chrome Safari iPhone

Problem:
In WebKit browsers, the Button widget in a GWT application seems to always act as a submit, regardless of event sinking (via ClickListener) behaviour added to it.

In Firefox (and IE!) the button works fine - that is the 'onclick' behaviour added to the widget via the GWT listening architecture is executed and the page is not submitted.

Solution:
A clue to the solution is in the code for the GWT Button (thank goodness for source code - see Button.adjustType(Element button)).

It seems to be a work around for Firefox. The W3C spec states that the default value for button type is 'submit'. Firefox does this explicitly in the DOM and when detected, this is fixed by the GWT JSNI (JavaScript Native Interface) code.

It would seem that for WebKit browsers this default is enforced but not made explicit in the DOM so this GWT snippet does not get a chance to resolve the issue. The work around is to always be explicit about the button type (as recommended in W3Schoools).

You do this in GWT as follows:

     Button b = new Button("click here");
     DOM.setElementAttribute(b.getElement(), "type", "button");

Redirect from a JSP page (to another server)

Keywords:
best practice tips JSP redirect XML

Problem:
<jsp:forward> is not for redirecting to an external server (and the address you forward to will have access the request attributes and parameters, which may be desirable, see discussion here: when to use response redirect and jsp forward.

Is using response.sendRedirect(...) ok/best-practice?
Solution:
The short answer, is it's fine. In summary, there are four approaches I can think of:

Approach #1: Scriptlet to sendRedirect

<%
    response.sendRedirect("http://www.example.com");
%>

This has worked since the introduction of JSPs (see: Implementing a Redirect in a JSP Page) ... sometimes you need the servlet API in your JSP page either because there's other content (eg HTML) that makes it reasonable not to be a pure servlet. Other times it's just convenient to have a text file compiled on the fly.

Though there are many that would disagree (Sciptlet snobs? see How to redirect a page in JSP - I don't see why you'd get wound up about scriptlets. Why would you write a servlet when one line of code in a text file gives you the same result?).

Approach #2: Use Apache JSTL and the c:redirect tag

<c:redirect url="http://www.example.com"/>

or scriptlets again:

<c:redirect url="<%=scriptlet logic%>"/>

Sure, it's 'pure' XML but you'll need the taglib definition in the JSP file and include the JSTL jars in your web application - as discussed in the notes for a previous post.

Approach #3: Refresh meta tag

You could get the JSP to produce a HTML page that contains the Refresh meta tag.

<html>
<head>
  <meta http-equiv="Refresh" content="0; url=http://www.example.com/">
</head>
<body></body>
</html>

Where 0 is the delay in seconds. If more than zero, you'd probably want to put some text on the page to explain what's happening.

Approach #4: Javascript

As above, you could get the JSP to produce a HTML page that contains javascript to perform the redirect.

<html>
<head>
<script type="text/javascript">
window.location = "http://www.example.com/";
</script>
</head>
<body></body>
</html>

You could use setTimeout to get the redirect to happen after a period of milliseconds.


Notes:
Out of curiosity I checked TCP monitor for what the browser is actually receiving when you use Approach #1 or #2 (sendRedirect or c:redirect)

HTTP/1.1 302 Moved Temporarily
Location: http://www.example.com

So it saves you from 2 lines of servlet (or scriptlet) code:

response.setStatus(302);
response.setHeader("Location", "http://www.example.com");

So the difference to Approaches #3 & #4 isn't great or much more inefficient really, in all cases it's up to the browser to go to the specified address.

Comments in JSON

Keywords:
comments JSON javascript

Problem:
Can you have comments in JSON?

Solution:
A discussion leading to the answer is on the Bytes IT forum.

Basically, though it's not defined in the JSON Grammar - on json.org - you can use slash-star /* ... */ comments to get most(?) javascript engines to ignore the comment text.

For example:

<script type="text/javascript">
    var nested_arrays = {
        cities: [
            ["Sydney", "Melbourne", "Canberra"]  /* Australia */
          , ["London", "Birmingham"]             /* UK */
          , ["Los Angeles", "New York"]          /* US */
        ]
    };
</script>

get/decode/unescape parameters from URL using Javascript

Keywords:
GET decode unescape request parameter param URL javascript

Problem:
You need to get at the (GET) parameters in the URL of the window with some client-slide javascript. There's no built in mechanism to access these by name in an unencoded way so it would seem you have to parse the window location ... this must have been done by some one before.

Solution:
Found two good blogs that discuss this:

1. Get URL Parameters Using Javascript - uses regular expressions


2. Get parameters from URL with JavaScript - uses substrings from the location.search (ie the query portion of the URL)

I found the second approach easier to read, but both miss unescaping the parameter value once obtained ... there's a built in function for that. So here's what I've used:

<script type="text/javascript">
/*
 * Retrieves the query parameter with the given parameter name or empty 
 * string if not defined
 * 
 * @param parameter name of the parameter to retrieve
 */
function queryParam(parameter) {
    var searchString = top.location.search;
    searchString = searchString.substring(1);// ommit the leading '?'

    var paramValue = '';
    var params = searchString.split('&');
    for (i=0; i<params.length;i++) {
        var paramPair = params[i];
        var eqlIndex = paramPair.indexOf('=');
        var paramName = paramPair.substring(0,eqlIndex);
        
        if (paramName == parameter) {
            paramValue = unescape(paramPair.substring(eqlIndex+1));
            return paramValue;
        }
    }
    return paramValue;
}
</script>

So from a URL http://www.example.com/formprocess?name=Fred%20Bloggs&date=01%2f01%2f2008
You can get the unescaped parameter values via:

var name = queryParam('name'); // 'Fred Bloggs'
var date = queryParam('date'); // '01/01/2008' 

Notes:
This doesn't handle multi-valued parameters ... the first (regex) blog post has some comments on how to handle this.

Firefox does not reflect input form field values via innerHTML

Keywords:
firefox innerHTML form input field value text select textarea

Problem:
In IE (and therefore the GWT test shell running on windows) you can call innerHTML on a DOM object and where it contains HTTP form input fields you will see an up-to-date reflection of the inputs from the user.

On firefox you simply get a reflection of the DOM as it was originally served up to the user.

This is a problem if you want to move HTML around and not loose the inputs already made by the user.

Eg: On clicking the button in firefox you won't see the input entered in the text box.

<form action="" method="get">       
    <SPAN id="MyContent">           
        <input type="text" name="textField" value="" /><br/>
    </SPAN>
</form>           
  
<button onClick="window.alert(MyContent.innerHTML);">discover user input</button>

Solution:
Found the solution on comp.lang.javascript - Firefox does not reflect selected option via innerHTML. I've extended the example code from this post to handle checkbox, radio and textarea ...

The idea is, every input field on the page must have an onBlur="updateDOM(this)" event handler, forcing the DOM to be updated and reflect the user's input.

<script type="text/javascript">
//
// Will be called by input fields when in 'update DOM' mode. This will
// make sure that changes to input fields in the form will be captured
// in the DOM - not necessary in IE but is required in Moz, etc as the DOM
// will otherwise reflect the page as it was initially.
//
// inputField : the input field that has just been tabbed out of (onBlur) OR the ID of the input field
function updateDOM(inputField) {
    // if the inputField ID string has been passed in, get the inputField object
    if (typeof inputField == "string") {
        inputField = document.getElementById(inputField);
    }
    
    if (inputField.type == "select-one") {
        for (var i=0; i<inputField.options.length; i++) {
            if (i == inputField.selectedIndex) {    
                inputField.options[i].setAttribute("selected","selected");
            } else {
                inputField.options[i].removeAttribute("selected");
            }
        }
    } else if (inputField.type == "select-multiple") {
        for (var i=0; i<inputField.options.length; i++) {
            if (inputField.options[i].selected) {
                inputField.options[i].setAttribute("selected","selected");
            } else {
                inputField.options[i].removeAttribute("selected");
            }
        }
    } else if (inputField.type == "text") {
        inputField.setAttribute("value",inputField.value);
    } else if (inputField.type == "textarea") {
        var text = inputField.value;
        inputField.innerHTML = text;
        inputField.setAttribute("value", text);
    } else if (inputField.type == "checkbox") {
        if (inputField.checked) {
            inputField.setAttribute("checked","checked");
        } else {
            inputField.removeAttribute("checked");
        }
    } else if (inputField.type == "radio") {
        var radioNames = document.getElementsByName(inputField.name);
        for(var i=0; i < radioNames.length; i++) {
            if (radioNames[i].checked) {
                radioNames[i].setAttribute("checked","checked");
            } else {
                radioNames[i].removeAttribute("checked");
            }
        }
    }
}
</script>

<form action="" method="get">        
    <SPAN id="MyContent">            
        <input type="text" name="textField" value="" onBlur="updateDOM(this)"/><br/>
    </SPAN>
</form>            
    
<button onClick="window.alert(MyContent.innerHTML);">discover user input</button>

Notes:
It gets slightly trickier if you have input fields that don't get filled in by the user - eg a date picker dropdown, which will set the textbox with the date for the user, hence they never click in the box and trigger the 'onBlur'. In this case, you'd put the onBlur event on the date picker button

Eg:

<input type="text" name="dateField" id="dateField" value="" onBlur="updateDOM(this)"/>
<button id="myDatePicker" onClick="... do datepicking stuff ..." onBlur="updateDOM('dateField')">
 ... date picking image ...
</button>

Post updated (Thu, 25 Mar 2010): With thanks to the helpful commentors, the above script incorporates better handling for textArea & radio fields as well as an issue I came across for 'select-multiple' (was missing from the original script). It should work fine with the update via document or form approaches discussed in the comments - as opposed to the onBlur which continues to be good enough for my usage.

Post updated (Fri, 5 Apr 2013): Beware trying to set innerHTML text with newlines - IE will lose them. Work around is set the value attribute afterwards and this will honour the newlines (and is meaningless for other browsers).

IE ignores inline script added to DOM

Keywords:
internet explorer GWT DHTML AJAX inline javascript "Object expected"

Problem:
I've got javascript code (GWT in this case) that adds an inline script to the DOM:

   <input ... onClick="myFunction()"/>
   <script type="text/javascript">
       function myFunction() {
           ... do stuff
       }
   </script>

If this code was on the page from the beginning, the event call to the function works fine. When this HTML is added to the page after it's loaded it appears the function call from the event fails.

IE Error (from the bottom left "! Done" in the status bar):

Line: 1
Char: 1
Error: Object expected

Further frustration - it works fine in Firefox.

Solution:
Aside: I haven't actually worked out why, but in GWT, if you use the method in a FlexTable setHTML the javascript will also not work in Firefox - you need to use setWidget(..).

Internet Explorer will not compile inline javascript code added to the DOM after the page has been loaded. What IE is trying to tell me in its cryptic error message is it's trying to call the function but can't find it.


Notes:
A good example of how to "smuggle" javascript into the DOM via img onLoad - Have Your DOM and Script It Too.