Propagating events through element layers as rendered in page

I am currently working on a project involving Mapbox and Leaflet.  As part of a performance enhancement feature I was implementing I came across an interesting issue on event handling in the map layers.

I am using D3 to implement a light-weight custom Leaflet layer to better handle large FeatureCollections.  From the server I receive a GeoJSON FeatureCollection of features that includes shapes, markers, and text that I need to render to the map.  In my custom layer the different feature types in the collection need to be placed into different map panes (overlay, markers, and popup panes) to maintain the proper layering.  These “panes” are implemented as sibling DIVs as oppose to being in a tree layout.

Depending on the purpose of the map, any one of these layers can be enabled to accept click events (and possibly more than one).  The issue I encountered was that only the top layer would actually accept the click — this layer typically being the marker or text.  If that layer was deemed “un-clickable” then nothing would happen because the event propagated up the DOM tree and not to the sibling “panes.”

What I need to handle is passing this event to all of the sibling panes to see who could handle it.  But I cannot just broadcast it!  I need to pass it to the panes in the order they are rendered on the page and only to each pane’s feature located at the exact mouse-click position.

Some background…

In mapbox maps there is an overlay pane for adding new overlay shapes, like polygons.  There is also the marker pane, for things like “pins” on point locations.  There are still other panes, like “popups” for additional items.  Because these panes are sibling divs and not hierarchical event handling can be tricky.

The FeatureCollection I receive can contain features mapping to 1..n of these panes.  So what I can end up with is multiple overlapping features on a map — e.g., a shape, a marker (or more) on that shape, and a text label on that marker.  These features know nothing about their layer or other features, but do contain information in their properties so  I can determine if the feature is clickable.

Because the layers are actually laid out in the DOM as absolute-positioned sibling ‘div’ elements I cannot use the browser’s event propagation.   What I need for my situation is information about the stack of elements as layered on the page by the renderer at the X/Y point of the mouse click.  An example of what i mean by the “layers” of the DOM elements is nicely displayed in this picture:

Layers on the google home page.
Element layers as rendered on the google home page.

I took this screen shot using a cool firefox plugin called Tilt, although I found out after I installed it that the firefox dev tools already has it built in as “3D View.”   Because everyone is familiar with it, I used Google’s homepage to illustrate stacked DOM element layers as I am working with such layers on the map.  The actual map did not illustrate this as well since the Tilt plugin was not splitting out SVG trees into layers.

To propagate the event the way  I need to do it, I need to dig through these layers top-to-bottom at the coordinates of the mouse click regardless of how these layers are represented in the DOM tree — CSS can move them anywhere.  Its like driving a skewer through the page at the point of click and then peeling back the shish kabob of features until i get to the one that can process the event.

A solution emerges…

It turns out there is a very useful document function for this called document.elementFromPoint(x,y).  This function will take an X/Y coordinate and return the element first encountered at those coordinates.  This is great, but I already know the first encountered element… it’s the one in the event that is triggered when the mouse click occurs.  So this alone does not help much, especially since it is only available on document and not sub-elements in order to process sub-trees.  But, it is a start…

So digging deeper I found that I could manipulate the results of the elementFromPoint function using very simple CSS!  When an element has the style “pointer-events: none,” click events pass through the element to what ever is next behind it.  This style alone could solve this issue sometimes if you are able to throw it around multiple places, but in the case of creating a custom layer I need to be more surgical so i don’t affect other layers that may be added to the map.  When this style is on an element, it is also skipped by the elementFromPoint function!  

Aside: Another page on quirksmode says you can do this with “display: none” as well, but that would change the page visually…  

A happy merging of these two ideas and a dose of recursion solves the problem.  All of the features I add to the page have a click handler that routes the click to a special function that does the following:

  1. Get the elementFromPoint from the X/Y coordinates in the mouse event.
  2. Check if this element is
    1. one of my features, which means it contains data (i.e., a “__data__” property) and has a key/value attribute I put on the element specifically for matching.
    2. is clickable (a separate function to determine this based on whatever…)
    3. is a child of some baseline element that I know all of my elements are inside.  This is to fail faster instead of parsing a lot of the page I don’t care about.  The base element for me, inside the mapbox map, is an element classed with ‘.leaflet-objects-pane’.
  3. If it all tests in #2 pass, return this element and pass it and it’s data (remember in D3 this is the element.__data__) off to the real click handler.
  4. If the tests in #2 fail, either return null (if i’m outside of my base element) or add the “pointer-events: none” style to the current element and call this function again recursively.   The current element is added to an array of “passed over elements” so that we can come back later and remove this style.  In order to not break the page, maintain the prior value of this style if there was one.

This loop continues until we either find an element that will handle the event or we are outside of the base element.

The last important step before returning from this function is to iterate over the “passed over elements” array and revert the “pointer-events: none” style.

Implementation
Here is the implementation as a member in the custom layer class, a little generalized for clarity.

        _digForElements: function keepDigging(evt, passedOverElementsFromLast) {
            var layer = this,
                  jMapPane = jQuery('.leaflet-objects-pane');
            if (!jMapPane) return null;

            var baseElement = jMapPane[0]
                attr = 'data-d3layerkey',
                attrVal = layer._layerId,
                passedOverElements = passedOverElementsFromLast || [],
                firstFound,
                element,
                jElement,
                existingCssValue;

            element = document.elementFromPoint(evt.x, evt.y);
            if (element && jQuery.contains(baseElement, element)) {
                jElement = jQuery(element);

                if (jElement.attr(attr) && jElement.attr(attr) === attrVal &&
                    _.isObject(element.__data__) && layer.options.isSeriesClickable(element.__data__)) {
                    firstFound = element;
                } else {
                    existingCssValue = jElement.css('pointer-events');
                    passedOverElements.push({
                        element: element,
                        previousCSSValue: existingCssValue === 'auto' ? '' : existingCssValue;
                    });

                    // Add style to it so that elementFromPoint bypasses it next time.
                    jElement.css({
                        'pointer-events': 'none'
                    });

                    firstFound = keepDigging.call(this, evt, passedOverElements);
                }
            }

            // clean up all elements we added classes to.  Also, end a sentence with a preposition.  done.
            var info;
            while (passedOverElements.length > 0) {
                // It did not have it before or it would not have come up in the search...
                // so remove it again.
                info = passedOverElements.pop();
                jQuery(info.element).css({
                    'pointer-events': info.previousCSSValue
                });
            }

            return firstFound;
        },
Share

JSON Parsing – Unexpected Token

This is a quick note because I just had this issue parsing JSON that was returned to me from the server. I was attempting to use JSON.parse, jQuery.parseJSON, and eval()… but i kept getting a range of errors depending how i attempted to manipulate it:

  • Unexpected Token {
  • Unexpected Token :
  • Unexpected Token ILLEGAL

And one or two more.  I searched a bunch of pages and it seems other people have this as well.  Also, my JSON validated with this validator.

Anyway, the solution for me, and others it seems, was to replace newline characters.  I was able to dig into the error a little bit using Chrome’s developer tools console to try to get the string to parse. The error allowed for expanding and hidden deeper in the stack trace was an error that suggested there were newline characters.  I also saw another post where someone had strings ending with the C string terminator ‘\0’

So, if you’re having this issue, try to get the JSON to parse in the console and see if there is some random character put in there that needs to be replaced.

success: function(response) {
   // REPONSE data has newlines!!  replace or it won't parse!
   var data = $.parseJSON(response.replace(/\n/g,"")); 
   // or use JSON.parse(...) if you're not using jquery and have a modern browser)
   // or... go download json2.js and use that.
   ...
                    
}

One other note… i’m using jsonp through jQuery so the data is returned as a string. Otherwise jQuery would have taken care of parsing it for me.

Share

Widgets in GWT Cell Tables

I have recently had the need to embed widgets into a GWT cell table.  I have read about the flyweight pattern, and specifically why cell tables are designed to not have widgets in them, but… when you have a requirement to complete you just need to do it.

I went through two iterations of this design and I am pretty happy with the final design.  I took some ideas from GWT Cell Widgets like EditTextCell as well as some experience from the first not-so-great implementation i created.  If you are new to Cells, it is important to realize that the Cell is instantiated one time, but called (with different data) for each row that is rendered.  This is why it is necessary to maintain a map of this cells data.

Here are some design ideas for this “WidgetCell”

  • Widget creation is expensive, so a widget for each row is only created one time and then stored in a cache for reuse.  This helps with the speed of the table rendering.
  • Widgets are never attached to the DOM.  They exist to be manipulated in memory and then have their HTML extracted and pushed into the DOM.
  • Events from the Widgets, therefore, are not handled.  If you need them you will need to change this implementation a little and pass them through the cell (onBrowserEvent method).  Cell events ARE handled (or can be in subclasses), which was enough for my purposes.
  • Finally, the widgets that I am using need to load data from the backend, so this “WidgetCell” populates the cells asynchronously.  This helps speed up initial table rendering as well.

One important thing to note on that last bullet: Browsers have a limit on the number of HTTP requests that can be active at a given time.  Additional requests (from my experience, i didn’t look into it deeply) are getting queued and block the UI from any other HTTP requests until they are cleared from the queue.  This means anything, like navigation, that requires an HTTP request will be queued and block the user interface.  The best solution I have found for this is to pass all related HTTP requests, like the ones for this widget, through a common RPC handler that performs queuing.  This can limit that data providers allowable HTTP requests, leaves additional HTTP connections available, and allows for cancelling uncalled or in-process requests if navigation changes prior to the table completely populating.

This WidgetCell is an abstract class.  It is meant to be extended, with the implementation defining the widget and handling the widget’s data.  The data can be stored in a map with an entry per row.  The abstract methods all pass through either the CellContext, the Cell context’s key, or the Cell index, depending on that is needed for that method.

Important things to remember:

  1.  As I mentioned earlier, this is a method for displaying widgets into Cells.  The widget lifecycle is NOT respected.  There is no attaching and detaching, and events are not handled from the widget.
  2. Widgets that change themselves over their lifetime will NOT have those changes reflected in the cell unless you take special care to manually reflect those changes in the DOM.  An example is a progress bar.  It will be rendered to the DOM, but any updates need to be re-rendered to the DOM manually because the widget is not attached.

Here is the code.  I am sure there will be questions that i can handle in the comments.  Additionally, I need to thank Colin Alworth at Sencha for comments and pointing out pitfalls that helped direct this final design, and David Maddison’s The GWT rendering process post for more insight on the widget lifecycle.

Here is a file with the code: Textfile with WidgetCell Code


import java.util.HashMap;
import java.util.Map;
import java.util.Map.Entry;

import com.google.gwt.cell.client.AbstractCell;
import com.google.gwt.cell.client.EditTextCell;
import com.google.gwt.cell.client.ValueUpdater;
import com.google.gwt.core.client.GWT;
import com.google.gwt.core.client.Scheduler;
import com.google.gwt.core.client.Scheduler.ScheduledCommand;
import com.google.gwt.dom.client.Element;
import com.google.gwt.dom.client.NativeEvent;
import com.google.gwt.safehtml.client.SafeHtmlTemplates;
import com.google.gwt.safehtml.shared.SafeHtml;
import com.google.gwt.safehtml.shared.SafeHtmlBuilder;
import com.google.gwt.user.client.Command;
import com.google.gwt.user.client.ui.RequiresResize;
import com.google.gwt.user.client.ui.Widget;

/**
 * A Cell for adding Widgets into CellTables. A single instance of a cell is used for a column, with the render method
 * being called for each new cell in a new row. data changes per row must come in through the render method (via the
 * setValue method in super)
 * 
 * The approached used here:
 * <ol>
 * <li>Keep a cache of widgets so only a finite number are made, and they are reused to minimize expensive widget
 * creation (this is based on the method GWT's {@link EditTextCell} uses to maintain data.)</li>
 * <li>The widgets are never attached to the DOM, they are only used for manipulating them and extracting their HTML</li>
 * <li>Events from the widgets, therefore, are not handled. Only events on the cells themselves. However, if needed, the
 * cell events can be passed along to the widgets in the reuse-map and handled. (the context contains the key for the
 * reuse-map)</li>
 * <li>For ajax use, only a wrapper (placeholder) is laid down in the DOM. This placeholder is replaced when the widget
 * HTML when it is ready.</li>
 * </ol>
 * 
 * @author mpickell
 * @version 1.0
 * @param <W>
 *            the widget that is to be used in this cell
 * @param <D>
 *            the type of data used for this widget. This is passed in through the render method for each cell created.
 * 
 * @see <b>For more information on this implementation:</b>
 *      http://www.sencha.com/forum/showthread.php?196112-Widget-Rendered-Grid/page2
 * 
 *      TODO: rename to AbstractClickableWidgetCell?
 * 
 */
public abstract class ClickableWidgetCell<W extends Widget, D> extends AbstractCell<D> implements RequiresResize {

    /**
     * Widget generator for use by this cell renderer. This used as a factory that is used when a new widget is needed.
     * 
     * @param <C>
     *            widget type returned
     */
    protected interface ClickableWidgetCellWidgetFactory<C extends Widget> {
        C createWidget();
    }

    private static int uniqueKeyForWidgetID = 0;

    /**
     * Widget's displaying the table may enlarge it. If this is non-null, it will be resized when all widgets display.
     */
    private RequiresResize containerToResizeAfterAllWidgetsDisplay;

    /**
     * map of widgets used in this column. Once created, the widget is cached so it doesn't need to be created again.
     * The key here is the context.getKey() object, which uniquely IDs a row. (reuse-map is based on design of GWT's
     * {@link EditTextCell})
     */
    private Map<Object, W> reusableWidgetsMap = new HashMap<Object, W>();
    private Map<Object, String> domPlaceholderIds = new HashMap<Object, String>();

    /**
     * This cell can be used as a non-clickable cell as well. this boolean will block the click and keydown events when
     * set to true via the setter.
     */
    private boolean blockClickableEvents = false;

    /**
     * The HTML templates used to render the cell placeholder.
     */
    interface ClickableWidgetCellPlaceholderTemplate extends SafeHtmlTemplates {
        /**
         * The template for the widget placeholder for this Cell, which includes styles and a value.
         * 
         * @param id
         *            the id for this placeholder template
         * @param value
         *            the safe value. Since the value type is {@link SafeHtml}, it will not be escaped before including
         *            it in the template. Alternatively, you could make the value type String, in which case the value
         *            would be escaped.
         * @return a {@link SafeHtml} instance of a simple container. The container position is set to relative so that
         *         any value placed inside can use absolute positioning.
         */
        @SafeHtmlTemplates.Template("<div style='position:relative;' id='{0}'>{1}</div>")
        SafeHtml cell(String id, SafeHtml value);
    }

    /**
     * Create a singleton instance of the templates used to render the cell.
     */
    private static ClickableWidgetCellPlaceholderTemplate placeholderTemplate = GWT
            .create(ClickableWidgetCellPlaceholderTemplate.class);

    /**
     * Construct a new ClickableWidgetCell
     */
    public ClickableWidgetCell() {
        super("click", "keydown");
    }

    /**
     * @return a factory that creates the default widget type, with any necessary setup, that is used by this Dto.
     */
    protected abstract ClickableWidgetCellWidgetFactory<W> getWidgetFactory();

    /**
     * Trigger any asynchronous calls that are needed to get data, if needed, and then update the widget. When update is
     * complete, execute command to push widget HTML to DOM.
     * 
     * @param widget
     *            the widget that needs to be updated
     * @param onDataReadyCallback
     *            the command to trigger when the widget is updated. This command displays the widget in the page
     */
    protected abstract void updateWidget(final W widget, final Context cellContext, final D data,
            final Command onDataReadyCallback);

    /**
     * After the widget is displayed, handle any clean up (like resize to fill cell maybe). This is a default impl.
     * 
     * @param widget
     *            the widget that was displayed in the DOM
     * @param cellContext
     *            this widget's context
     * @param data
     *            the data used when displaying this widget.
     */
    protected void finalizeWidgetAfterDisplayInGrid(final W widget, final Context cellContext, final D data) {
    }

    /**
     * When resize is triggered on this cell (passed down from the grid) the cell implementation can handle it. This is
     * the default impl.
     * 
     * @param widget
     *            a widget from the reuse queue
     * @param cellContextKey
     *            this widget's context
     * @param placeholderId
     *            the placeholder ID for this widget in the DOM, so the HTML can be updated.
     */
    protected void handleWidgetOnWindowResize(W widget, Object cellContextKey, String placeholderId) {
    }

    /**
     * Default implementation.
     * 
     * This method returns HTML to be placed in the page while waiting for the widget to be displayed. If the widget
     * requires service calls, this will be displayed until the service calls return and the onDataReadyCallback command
     * is executed.
     * 
     * @return a SafeHtml object containing whatever HTML should be displayed while the main widget is being updated.
     *         This will be completely removed and placed by the widget HTML when the updateWidget method calls the
     *         onDataReadyCallback.
     */
    protected SafeHtml getWidgetUpdatingHtml() {
        return new SafeHtml() {
            private static final long serialVersionUID = 1L;

            @Override
            public String asString() {
                return "<i>Loading...</i>";
            }
        };
    }

    @Override
    public void onBrowserEvent(Context context, Element parent, D widgetDataDto, NativeEvent event,
            ValueUpdater<D> valueUpdater) {
        super.onBrowserEvent(context, parent, widgetDataDto, event, valueUpdater); // Handles user press actual
                                                                                   // ENTER key
        if ("click".equals(event.getType())) {
            onEnterKeyDown(context, parent, widgetDataDto, event, valueUpdater);
        }
    }

    @Override
    protected void onEnterKeyDown(Context context, Element parent, D widgetDataDto, NativeEvent event,
            ValueUpdater<D> valueUpdater) {
        if (!blockClickableEvents && valueUpdater != null) {
            valueUpdater.update(widgetDataDto);
        }
    }

    @Override
    public void render(final Context context, final D widgetDataDto, final SafeHtmlBuilder sb) {
        /*
         * Widgets are reused so that their (expensive) creation is not performed each time. After creation, they are
         * attached and detached as needed and re-added to the page.
         */
        Object rowKey = context.getKey();

        /*
         * Either get the existing widget from the cache or create it.
         */
        W wfc = reusableWidgetsMap.get(rowKey);
        if (wfc == null) {
            // Create once and store.
            wfc = this.getWidgetFactory().createWidget();
            reusableWidgetsMap.put(rowKey, wfc);
        }
        final W wfcAsFinal = wfc;

        /*
         * Setup a unique key for the placeholder in this row
         */
        if (!domPlaceholderIds.containsKey(context.getKey())) {
            setupPlaceholderId(context);
        }

        /*
         * Allow the Dto to update the widget, possibly after asynchronous calls to service. Once updated, the command
         * is (i.e., has to be) executed to finalize the displaying of the cell.
         * 
         * This update is triggered as a GWT "thread" using the deferred command so that the render method can complete
         * quickly. The update and display of the widget can be deferred safely because everything needed for it is
         * packaged in the updateWidget method
         */
        Scheduler.get().scheduleDeferred(new ScheduledCommand() {
            @Override
            public void execute() {
                updateWidget(wfcAsFinal, context, widgetDataDto, new Command() {
                    @Override
                    public void execute() {
                        ClickableWidgetCell.this.displayWidget(context, widgetDataDto);
                    }
                });
            }
        });

        /*
         * Put the placeholder in the table so we have a location to go to later. Put the "busy" html in there.
         */
        sb.append(placeholderTemplate.cell(domPlaceholderIds.get(context.getKey()), getWidgetUpdatingHtml()));
    }

    private void displayWidget(final Context cellContext, final D widgetDataDto) {
        Element e = getElementsBySelectorJSNI("#" + domPlaceholderIds.get(cellContext.getKey()));
        if (e != null && reusableWidgetsMap.containsKey(cellContext.getKey())
                && reusableWidgetsMap.get(cellContext.getKey()).getElement() != null) {
            e.setInnerHTML(reusableWidgetsMap.get(cellContext.getKey()).getElement().getString());

            /*
             * The widget is now displayed. Allow the implementation to perform any adjustments to it. The
             * implementation can use the domPlaceholderIds map (via getter) to replace the HTML in the page since the
             * element in the widget is never attached to the page.
             * 
             * Changes to the widget will be reflected in the widget map, placeholder ID can be used (via getter, as a
             * string so it cannot be changed) to push any updated HTML to the page.
             */
            Scheduler.get().scheduleDeferred(new ScheduledCommand() {
                @Override
                public void execute() {
                    ClickableWidgetCell.this.finalizeWidgetAfterDisplayInGrid(reusableWidgetsMap.get(cellContext.getKey()),
                            cellContext, widgetDataDto);
                }
            });

            /*
             * This is a widget that might be expanding the table. Call on the resize container to resize.
             */
            if (containerToResizeAfterAllWidgetsDisplay != null) {
                containerToResizeAfterAllWidgetsDisplay.onResize();
            }
        }

    }

    /**
     * Query for elements using selectors
     * 
     * @param selectors
     *            a string containing one or more CSS selectors separated by commas
     * @return a JS array of elements.
     * 
     * @see https://developer.mozilla.org/en-US/docs/DOM/Document.querySelector
     * @see http://www.w3.org/TR/selectors-api/
     */
    protected native Element getElementsBySelectorJSNI(String selector) /*-{
		return $doc.querySelector(selector);
    }-*/;

    /**
     * Create a unique ID for a placeholder that will be placed in the cell on render.
     * 
     * @param context
     */
    private void setupPlaceholderId(Context context) {
        if (uniqueKeyForWidgetID >= Integer.MAX_VALUE) {
            // housekeeping.
            uniqueKeyForWidgetID = 0;
        } else {
            uniqueKeyForWidgetID++;
        }
        domPlaceholderIds.put(context.getKey(), "WidgetCellID_" + uniqueKeyForWidgetID);
    }

    /**
     * If true, this will block the clickable events and this cell will simply render the widget without accepting
     * events.
     * 
     * @param blockClickableEvents
     */
    public void setBlockClickableEvents(boolean blockClickableEvents) {
        this.blockClickableEvents = blockClickableEvents;
    }

    @Override
    public void onResize() {
        /*
         * on resize of the page, update the widget to the new cell size if necessary.
         */
        if (reusableWidgetsMap != null && !reusableWidgetsMap.isEmpty()) {
            for (Entry<Object, W> entry : reusableWidgetsMap.entrySet()) {
                ClickableWidgetCell.this.handleWidgetOnWindowResize(entry.getValue(), entry.getKey(),
                        domPlaceholderIds.get(entry.getKey()));
            }
        }
    }

    /**
     * Get the ID for the widget placeholder in the page. the innerHtml of this element is the widget HTML.
     * 
     * @param cellContextKey
     * @return
     */
    protected String getDomPlaceholderId(Object cellContextKey) {
        return domPlaceholderIds.get(cellContextKey);
    }

    /**
     * Set the container of this grid so it can be resized when all widgets display.
     * 
     * @param containerToResizeAfterAllWidgetsDisplay
     *            the container that implements RequiresResize & IsWidget
     */
    public void setContainerToResizeAfterAllWidgetsDisplay(RequiresResize containerToResizeAfterAllWidgetsDisplay) {
        this.containerToResizeAfterAllWidgetsDisplay = containerToResizeAfterAllWidgetsDisplay;
    }

}

Share

GWT & mvp4g: Displaying history on the client side.

I’m currently working on a task to display and persist history using GWT and mvp4g.  There are many ways of doing this, but here is how i’m going it.  Maybe this will help you with an additional approach, or maybe you can help me do it better.

Implementation Attempts

1st Pass: GWT’s History Event

First I attempted to use GWT’s History ValueChange event to capture changes happening in history.  I run into trouble here right away.  This event only seemed to capture history changes that occurred when I called History.newItem(...) myself; mvp4g was not triggering it.  I search around on the internet and didn’t find any explanation of what this event was supposed to be doing, or how mvp4g handled it, but apparently mvp4g was not triggering this event for its history changes.  This method was out.

UPDATE: For some reason it eluded me in my searches until after my final design, but you could create your own PlaceService, which is using the ValueChange event in order to re-create views from history.  There are a lot of options here, which i am going to take a look at because it will most likely change the design I am currently using.  See a write-up here.

2nd Pass: hashchange Event

I am lucky to be supporting only most recent browsers, so the next method I attempted was to use the hashchange event to capture every change made in the browser.  This worked very well and caught everything.

In the end, i did not use it because it is detached from the mvp4g History Converters.  It is cleaner to keep all of the history implementation in one place so each converter can define its own behavior, and that behavior is maintained along with those converters.  I would have had to catch a change and then query into one of the converters to get additional information on resolving descriptions, etc.   That is messy enough, but then there is the issue of the converters not being programmatically accessible (I couldn’t figure out how to get them from mvp4g… i would love to know if someone can tell me), and they do not have an eventBus instance.

Anyway, I did not end up using this approach in my design but here is that code if it can help you out.  At the very least, it was good to get to know this hashchange event.  GWT does not have any built in hooks to this event, so I created a new EventHandler to encapsulate it and make it injectable into wherever it is needed.

Here’s the class I created:

import com.google.gwt.core.client.Scheduler;
import com.google.gwt.core.client.Scheduler.ScheduledCommand;
import com.google.gwt.event.shared.EventHandler;
import com.google.gwt.event.shared.GwtEvent;
import com.google.gwt.event.shared.HandlerManager;
import com.google.gwt.event.shared.HandlerRegistration;
import com.google.gwt.event.shared.HasHandlers;
import com.google.inject.Singleton;
import com.mvp4g.client.event.BaseEventHandler;

/**
 * EventHandler for hash change events. This is an eventhandler so that it is a singleton that can be injected where
 * needed, and can access the eventbus if events are sent that way.
 */
@Singleton
@com.mvp4g.client.annotation.EventHandler
public class HashChangeEventEmitter extends BaseEventHandler implements HasHandlers {

 // handlers
 private HandlerManager handlerManager;

 protected HashChangeEventEmitter() {
 handlerManager = new HandlerManager(this);
 }

  @Override
 public void setEventBus(EventBus eventBus) {
 super.setEventBus(eventBus);

     Scheduler.get().scheduleDeferred(new ScheduledCommand() {
         @Override
         public void execute() {
             registerHashChangedEventListenerInDOM();
         }
     });
 }

 private void fireHashChangedEventFromDOM(String oldUrl, String newUrl) {
     fireHashChangedEvent(new HashChangedEvent(oldUrl, newUrl));
 }

 /**********************************************************
 * Event registration and emitting
 **********************************************************/

 private void fireHashChangedEvent(HashChangedEvent event) {
     if (handlerManager.getHandlerCount(HashChangedEvent.TYPE) > 0) {
         handlerManager.fireEvent(event);
     }
 }

 /** {@inheritDoc} */
 public HandlerRegistration addHashChangedEventHandler(HashChangedEventHandler handler) {
     return handlerManager.addHandler(HashChangedEvent.TYPE, handler);
 }

 @Override
 public void fireEvent(GwtEvent event) {
     handlerManager.fireEvent(event);
 }

 /**********************************************************
 * Event definition
 **********************************************************/
 public static interface HashChangedEventHandler extends EventHandler {
     void onHashChanged(HashChangedEvent event);
 }

 public static class HashChangedEvent extends GwtEvent {

 public static Type TYPE = new Type();

 private final String oldUrlHash;
 private final String oldUrlRaw;
 private final String newUrlHash;
 private final String newUrlRaw;

 public HashChangedEvent(String oldUrlRaw, String newUrlRaw) {
     super();
     this.oldUrlHash = extractHash(oldUrlRaw);
     this.newUrlHash = extractHash(newUrlRaw);

     this.oldUrlRaw = oldUrlRaw;
     this.newUrlRaw = newUrlRaw;
 }

 private String extractHash(String url) {
     return url.contains("#") ? url.substring(url.indexOf('#')) : "";
 }

 @Override
 public Type getAssociatedType() {
     return TYPE;
 }

 @Override
 protected void dispatch(HashChangedEventHandler handler) {
     handler.onHashChanged(this);
 }

 /** @return just the hash from the old URL */
 public String getOldUrlHash() {
     return oldUrlHash;
 }

 /** @return the entire old URL */
 public String getOldUrlRaw() {
     return oldUrlRaw;
 }

 /** @return just the hash from the new URL */
 public String getNewUrlHash() {
     return newUrlHash;
 }

 /** @return the entire new URL */
 public String getNewUrlRaw() {
     return newUrlRaw;
 }

 }

 private final native void registerHashChangedEventListenerInDOM()
/*-{
 var hashChangedEventEmitter = this;

   $wnd
    .addEventListener(
        "hashchange",
        function(event) {
       hashChangedEventEmitter.@<yourPathToEmitter>.HashChangeEventEmitter::fireHashChangedEventFromDOM(Ljava/lang/String;Ljava/lang/String;)(event.oldURL, event.newURL);
       });
 }-*/;

}

The event is a standard GwtEvent, and the class can be injected wherever it is needed like this:

    @Inject
    protected HashChangeEventEmitter hashChangeEventEmitter;

Final Solution: Instrumenting the History Converters

mvp4g handles history in GWT through History Converters, which uses the GWT History framework.  These allow you to attach history entries directly to EventBus calls.  Anytime you trigger a certain event, you can specify how to capture (and recreate) the resulting action in history.

The design I’m currently using creates an abstract history converter that contains a common method called at the end of every history converter implementation’s event bus call (the one creating the token to save for this history entry).  This method, in turn, calls a couple of abstract methods implemented in the history converters that resolve any information needed to display what this history entry is and the history token itself.  This information is then passed to the presenter of the history display view.  This instruments each history converter and requires all new history converters implement these methods so that the history display is kept updated.

The presenter for the history view is injected the same way the HashChangeEventEmitter was injected in the 2nd pass above.  This presenter then handles pushing the history information to the backend as well as updating the view.

Re-evaluation and design forthcoming

It eluded me in my original google searches when I was working on my original designs, but I am going to revisit this and look at implementing it in a cleaner way using a mvp4g Custom PlaceService.  This is the place that has a collection of all of the history converters and is a central point that can be used to implement exactly what I’m trying to do without needing to clutter up the history converters. It is exactly what I was look for, and knew was there somewhere, but missed…

I will update this post after that…

Share

Convert an old Scoop CMS to WordPress

I recently took on a project to move a site running on an old CMS called “Scoop” over to a new WordPress site. My original plan was to put the scoop site over on the new hosting site so that permalinks would forward past the wordpress site to the old site. I wouldn’t have to convert anything, and the new site would keep rolling. It turns out that the new hosting site does not support perl, which is the language of the Scoop CMS. Really… i had no idea what i was going to do so i started hunting…

I began looking at an extraction of the Scoop database given to me by Jeremy, who was awesome to work with (thanks Jeremy!).  I searched through wordpress and posted a couple of questions there (search for mpickell in the forum)… and their large list of importers/converters did not include one for this.  I did not get a lot of responses on my posts..   Long story short.. i wrote one in PHP and it is listed here — it works, i used it. I did edit it to make it prettier for this blog, let me know if something is not working right.

Since this was the first time i did something like this, and i was vaguely familiar with WordPress, I did not write an importer. This script extracts a Scoop CMS DB into a WordPress eXtended RSS (WXR) format ( a nice definition here ). So when you import this/these files, the importer to use is “WordPress”

A couple important notes:

  • You have to edit some variables at the beginning of this file to fill in your data.
  • There are two optional URL parameters (files and killAfter). Read the comments. Basically, you can split your export into multiple files so you have fewer problems ( see large import files and PHP.INI adjustments and here for memory issues. )
  • When you import split files, there is a problem with the 2nd, 3rd, 4th, etc. imports. You have to go through EVERY AUTHOR and select themseleves from the “or assign to existing user…” combo box. If you don’t, you LOSE the author link to every post — They all become “admin”. This is annoying. I posted to the wordpress forum and did not receive any response on this being an issue even when i mentioned the almost 500 authors i was working with… oh well, thank you jQuery
  • Turn on debugging. put the line define('WP_DEBUG', true); in your wp-config.php file. I think i read that there was more fine-grained debugging that could be enabled in the WordPress importer plugin code, but i didn’t end up needing that.
  • One last thing. You can fake the Scoop concept of “dairies” using a new user role and Pages. A new user role that enhances the “subscriber” role with the ability to create pages fits the need. I then added custom widgets to the site that automatically showed any new pages in reverse chronological order.

I think that’s it… i’ll update this post if I remember something i missed. I was going to write this in perl but i had trouble installing one of the modules for database access on my iMac. So i learned PHP, which i think turned out to be a much better decision anyway — although it leaves you with messy code, sorry.

<?php

/*
 *  I used MAMP as my testing system.  I received an extraction of the Scoop DB
 * in the form of an SQL file (i.e., create table statements, inserts, etc).  So I 
 * imported this into my local MySQL and worked on that copy of the data to create 
 * this script.
 *
 * I then used a local WordPress installation to test the imports
 */

// Don't allow a timeout... this is long!! 
ini_set('max_execution_time', 300); //300 seconds = 5 minutes

//Set up parameters
$mySqlUsername = "YourMySQLUsernameHere";
$mySqlPassword = "YourMySQLPasswordHere";
$mySqlHost = 'localhost';
$db_name = "YourScoopDatabaseNameHere!"; // This is the SCOOP DB name

// domain name you're switching to, as in "www.".$newDomainName.".com"
$newDomainName = 'mothertalkers'; 

print "Starting...";

/*
 * Attempt to get num of files from URL
 *
 * add something like "translate.php?files=10" to the URL to tell
 * this script how many different files to split the Scoop DB into.
 */
if (isset($_GET['files'])) {
  // This determines how to split the scoop stories into WXR import files
  $numOfWXRFilesToCreate = intval($_GET['files']);
} else {
  $numOfWXRFilesToCreate = 1;
}
print "files set ". isset($_GET['files']) . ", files is int " 
	. is_int($_GET['files']).", num is ".$_GET['files'];

// 
/*
 * If testing, kill after creating X number of files... 
 *
 * add something like "translate.php?killAfter=1" to the URL to tell
 * this script when to stop.. for when you are testing and don't want
 * to wait for the full extraction
 */
if (isset($_GET['killAfter'])) {
  // This determines how to split the scoop stories into WXR import files
  $killAfter = intval($_GET['killAfter']);
} else {
  $killAfter = -1;
}


print "Breaking scoop db into " . $numOfWXRFilesToCreate . " files.<br>";

// Open connection to MAMP mySql -- I used MAMP, this shouldn't matter.
// We need two connections for when we are pulling the posts and comments.
$dbConnection1 = mysql_connect($mySqlHost, $mySqlUsername, $mySqlPassword);
$dbConnection2 = mysql_connect($mySqlHost, $mySqlUsername, $mySqlPassword);

if (!$dbConnection1) {
    die('Not connected : ' . mysql_error($dbConnection1));
} else {
	print "Connection 1 to mysql is ok!<br>";
}
if (!$dbConnection2) {
    die('Not connected : ' . mysql_error($dbConnection2));
} else {
	print "Connection 2 to mysql is ok!<br>";
}


// Open connection to scoop database
$db_selected = mysql_select_db($db_name, $dbConnection1);
if (!$db_selected) {
    die ('Can\'t use foo : ' . mysql_error($dbConnection1));
} else {
	print 'Connected to ' . $db_name . '!<br>';
}
$db_selected2 = mysql_select_db($db_name, $dbConnection2);
if (!$db_selected) {
    die ('Can\'t use foo : ' . mysql_error($dbConnection2));
} else {
	print 'Connected to ' . $db_name . '!<br>';
}

// Prep for looping to create files
$count = getNumberOfScoopStories();
$postsPerFile = ((int) ($count / $numOfWXRFilesToCreate));
// $postsperFile count may leave a few stragglers at the end..
$remainderPosts = $count - ($postsPerFile * $numOfWXRFilesToCreate);

// increase posts per file by 1 if necessary to make sure we get everything
$postsPerFile += ($remainderPosts > 0) ? 1 : 0;
print "There will be " . $postsPerFile . " posts in each file.  The last will be smaller.<br>";

$loopVar = 0; // global so i don't have to rewrite all the writeToFile function calls.. multi-files was added late.

// count the processed posts... we might not need as many files.  
// This was for escaping early, as well as debug to make sure i got everything.
$processedPostsCount = 0;
$totalTagsWritten = 0;

// Loop to create separate files
for ($i=0; $i<$numOfWXRFilesToCreate; $i++) {
  if ($killAfter > 0 && $i >= $killAfter) {
    break; // quit now..
  }
  
  if ($processedPostsCount >= $count) {
    break; //quit early, we got everything (this doesn't happen anymore, i was counting incorrectly... )
  }
  
  
      $loopVar = $i;

      //start file -- always put the header
      writeToFile('<?xml version="1.0" encoding="UTF-8" ?>', 'w');
      writeToFile('<rss version="2.0"');
      writeToFile('xmlns:excerpt="http://wordpress.org/export/1.2/excerpt/"');
      writeToFile(	'xmlns:content="http://purl.org/rss/1.0/modules/content/"');
      writeToFile(	'xmlns:wfw="http://wellformedweb.org/CommentAPI/"');
      writeToFile(	'xmlns:dc="http://purl.org/dc/elements/1.1/"');
      writeToFile(	'xmlns:wp="http://wordpress.org/export/1.2/"');
      writeToFile('>');
      writeToFile('<channel>');
      writeToFile(	'<title>Mothertalkers test import</title>');
      writeToFile(	'<link>http://www.mothertalkers.com</link>');
      writeToFile(	'<description>a tagline</description>');
      writeToFile(	'<pubDate>Sat, 17 Mar 2012 23:22:58 +0000</pubDate>');
      writeToFile(	'<language>en</language>');
      writeToFile(	'<wp:wxr_version>1.2</wp:wxr_version>');
      writeToFile(	'<wp:base_site_url>http://mothertalkers.com/</wp:base_site_url>');
      writeToFile(	'<wp:base_blog_url>http://www.mothertalkers.com</wp:base_blog_url>');
      
      
	// These "output..." functions take a boolean left over from debugging

      /*****************************************************************
       *  FIRST, pull out all users who are authors
       *****************************************************************/
      outputAuthorXML();
      
      
      /*****************************************************************
       *  NEXT, Get the categories and tags
       *****************************************************************/
      outputTagsXML();
      
      
      /*****************************************************************
       *  NEXT, Add stories with comments
       *****************************************************************/
      outputStoriesXML(true, $i); // explicitly passed in so you know where it's used.
      
      // end file -- always put the closing tags
      writeToFile("\n\n" . "</channel> </rss>");

} // end of main loop

print $totalTagsWritten . " tags were written from this DB<br>";

print "<br><br>boo-yah!  done!";

/*****************************************************************
 *  Helper functions
 *****************************************************************/

// Get a count of the stories in the SCOOP DB
function getNumberOfScoopStories() {
  global $dbConnection1;

  $returnVal = -1;

	// This WHERE clause must match the where clause in outputStoriesXML!
    $query = 'Select count(*) as count from stories WHERE displaystatus in (0,1)';
    
    $result = mysql_query($query,$dbConnection1);
    if (!$result) {
        $message  = 'Invalid COUNT query: ' . mysql_error($dbConnection1) . "\r";
        $message .= 'Whole COUNT query: ' . $query;
        die($message);
      }
    
    // output the tags
    while ($row = mysql_fetch_assoc($result)) {
      $returnVal = $row['count'];
    }
    mysql_free_result($result);
    
    print "Scoop DB contains " . $returnVal . " posts!";

  return $returnVal;
}

/* 
 * This takes a string and makes it a nicename to match
 * how scoop did it.  (this was tested.. )
 *
 * (Jeremy @ dailykos gave me the way "scoop" does it in perl so i could match it here)
 */
function makeScoopStyleNiceName($text, $useLeadingDash = true, $sanitize = true)
{
  // build patterns
  $patterns = array("/ /", "/&(.*?);/", "/\?/", '/\"/', "/=/", "/&/", "/%/", "/-{2,}/", "/~/", "/\//");
  $replacements = array("-", "", "", "", "", "", "", "", "", "-");

  $var = preg_replace($patterns, $replacements, $text);
  
  if ($sanitize) {
      return ($useLeadingDash ? '-' : '') . urlencode($var);
    } else {
      return ($useLeadingDash ? '-' : '') . $var;
    }
}

/*
 * Extract the authors... i left in the comments from wherever i 
 * copied this from... 
 */
function outputAuthorXML($showAuthorOutput = true) {
  global $dbConnection1, $dbConnection2;
  // Formulate Query
    // This is the best way to perform an SQL query
    // For more examples, see mysql_real_escape_string()
    /*
    EXAMPLE
    $query = sprintf("SELECT firstname, lastname, address, age FROM friends 
    WHERE firstname='%s' AND lastname='%s'",
    mysql_real_escape_string($firstname),
        mysql_real_escape_string($lastname));
    */
    
    // only pull users who have stories... 
    $query = "Select distinct uid, nickname from users where uid in (select distinct aid from stories)";
    // Perform Query
    $result = mysql_query($query, $dbConnection1);
    // This shows the actual query sent to MySQL, and the error. Useful for debugging.
    if (!$result) {
        $message  = 'Invalid AUTHORS query: ' . mysql_error($dbConnection1) . "\r";
        $message .= 'Whole AUTHORS query: ' . $query;
        die($message);
      }
    // Use result
    // Attempting to print $result won't allow access to information in the resource
    // One of the mysql result functions must be used
    // See also mysql_result(), mysql_fetch_array(), mysql_fetch_row(), etc.
    while ($row = mysql_fetch_assoc($result)) {
        $newNickName = changeNickName($row['nickname']);
        
        $output = "<wp:author><wp:author_id>".$row['uid']."</wp:author_id><wp:author_login>".$newNickName."</wp:author_login><wp:author_email></wp:author_email><wp:author_display_name>".wxr_cdata($newNickName)."</wp:author_display_name><wp:author_first_name>".wxr_cdata($newNickName)."</wp:author_first_name><wp:author_last_name>".wxr_cdata(" ")."</wp:author_last_name></wp:author>";
        if ($showAuthorOutput) {
          //echo htmlspecialchars($output, ENT_QUOTES);
          //echo "<br>";
          writeToFile($output);
        }
     
    }
    // Free the resources associated with the result set
    // This is done automatically at the end of the script
    mysql_free_result($result);
}

function outputTagsXML($showCategoriesAndTagsOutput = true) {
  global $dbConnection1, $dbConnection2;

  // pull tags.  Scoop doesn't have categories, so all tags will be from 'uncategorized'
    $query = "Select distinct tag from story_tags";
    $result = mysql_query($query,$dbConnection1);
    if (!$result) {
        $message  = 'Invalid CAT/TAGS query: ' . mysql_error($dbConnection1) . "\r";
        $message .= 'Whole CAT/TAGS query: ' . $query;
        die($message);
      }
    
    // Print out the category
    if ($showCategoriesAndTagsOutput) {
      $output = "<wp:category><wp:term_id>1</wp:term_id><wp:category_nicename>uncategorized</wp:category_nicename><wp:category_parent></wp:category_parent><wp:cat_name>".wxr_cdata("Uncategorized")."</wp:cat_name></wp:category>";
      //echo htmlspecialchars($output, ENT_QUOTES);
      writeToFile($output);
    }
    
    // output the tags
    while ($row = mysql_fetch_assoc($result)) {
        $tag = $row['tag'];
        $niceTag = makeScoopStyleNiceName($tag, false);
        
        $output = "<wp:tag><wp:tag_slug>".$niceTag."</wp:tag_slug><wp:tag_name>".wxr_cdata($tag)."</wp:tag_name></wp:tag>";
        if ($showCategoriesAndTagsOutput) {
          //echo htmlspecialchars($output, ENT_QUOTES);
          //echo "<br>";
          writeToFile($output);
        }
     
    }
    // Free the resources associated with the result set
    // This is done automatically at the end of the script
    mysql_free_result($result);
 
}

function outputStoriesXML($showStoriesOutput = true, $loopVar) {
  global $dbConnection1, $dbConnection2, $postsPerFile, $processedPostsCount, $numOfWXRFilesToCreate, $newDomainName;
  // pull tags.  Scoop doesn't have categories, so all tags will be from 'uncategorized'
    $query = 'Select s.title, s.sid, s.dsid, s.id, UNIX_TIMESTAMP(s.time) as time, u.nickname, s.aid, s.introtext, s.bodytext, s.commentstatus FROM stories s, users u ' .
             'WHERE u.uid=s.aid AND s.displaystatus in (0,1) '.
             'LIMIT ' . ($postsPerFile * $loopVar) . ', ' . $postsPerFile;//((($numOfWXRFilesToCreate-1) == $loopVar) ? '' : ', ' . $postsPerFile);
    
    $result = mysql_query($query,$dbConnection1);
    if (!$result) {
        $message  = 'Invalid STORIES query: ' . mysql_error($dbConnection1) . "\r";
        $message .= 'Whole STORIES query: ' . $query;
        die($message);
      }
    
    // output the tags
    $postsInThisFile = 0;
    while ($row = mysql_fetch_assoc($result)) {
      $processedPostsCount++; 
      $postsInThisFile++;
      
      // Get the dates for the posts
      //ex: Wed, 07 Jan 2009 17:30:08 +0000
      $publishDate = date("D, d M Y H:i:s O", $row['time']);
      //ex: 2009-01-07 17:30:08
      $postDate_gmt = date("Y-m-d H:i:s", $row['time']);
      
      
      $sidSplit = preg_split("/\//", $row['sid'], -1, PREG_SPLIT_OFFSET_CAPTURE);
      $link = "http://www.".$newDomainName.".com/story/".$sidSplit[0][0]."/".$sidSplit[1][0]."/".$sidSplit[2][0]."/".$row['id']."/" . makeScoopStyleNiceName($row['title']);
      //echo $link . "<br>";
      
        writeToFile( '	<item>' .
                      '<title>'.fixOldHtmlCodes($row['title']).'</title>' .
                      '<link>'.$link.'</link>' .
                      '<pubDate>'.$publishDate.'</pubDate>' .
                      '<dc:creator>'.changeNickName($row['nickname']).'</dc:creator>' .
                      '<guid isPermaLink="false">'.$link.'</guid>' .
                      '<description></description>' .
                      '<content:encoded>'.wxr_cdata($row['introtext'] . (IsNullOrEmptyString($row['bodytext']) ? '' : '<br/>') .$row['bodytext']).'</content:encoded>' .
                      '<excerpt:encoded>'.wxr_cdata(" ").'</excerpt:encoded>' .
                      '<wp:post_id>'.$row['id'] .'</wp:post_id>' .
                      '<wp:post_date>'.$postDate_gmt.'</wp:post_date>' .
                      '<wp:post_date_gmt>'.$postDate_gmt.'</wp:post_date_gmt>' .
                      '<wp:comment_status>'.($row['commentstatus'] == 0 ? 'open' : 'closed').'</wp:comment_status>' .
                      '<wp:ping_status>open</wp:ping_status>' .
                      '<wp:post_name>'.makeScoopStyleNiceName($row['title']).'</wp:post_name>' .
                      '<wp:status>publish</wp:status>' . // because we're only bringing in published stories
                      '<wp:post_parent>0</wp:post_parent>' .
                      '<wp:menu_order>0</wp:menu_order>' .
                      '<wp:post_type>post</wp:post_type>' .
                      '<wp:post_password></wp:post_password>' .
                      '<wp:is_sticky>0</wp:is_sticky>' .
                      '<category domain="category" nicename="uncategorized">'.wxr_cdata("Uncategorized").'</category>');
            insertTags($row['sid'], $row['id']);
            writeToFile('<wp:postmeta>' .
                      '  <wp:meta_key>_edit_last</wp:meta_key>' .
                      '  <wp:meta_value>'.wxr_cdata($row['aid']).'</wp:meta_value>' .
                      '</wp:postmeta>' );
            insertComments($row['id']) ;
            writeToFile('</item>');
        if ($showStoriesOutput) {
          //echo htmlspecialchars($output, ENT_QUOTES);
          //echo "<br>";
          //writeToFile($output . "\r");
        }
    }
    // Free the resources associated with the result set
    // This is done automatically at the end of the script
    mysql_free_result($result);
    
    print $postsInThisFile . " posts added to file " . ($loopVar + 1) . "<br>";
}

function insertTags($story_sid, $id) {
  global $dbConnection1, $totalTagsWritten;
    // Get tags for this story.
    $query = 'Select tag FROM story_tags where sid=\''.$story_sid.'\' order by tag_order';
    //print $query . "<br>";
    
    $result = mysql_query($query,$dbConnection1);
    if (!$result) {
        $message  = 'Invalid STORY TAGS query: ' . mysql_error($dbConnection1) . "\r";
        $message .= 'Whole STORY TAGS query: ' . $query;
        die($message);
      }
    
    // output the tags
    while ($row = mysql_fetch_assoc($result)) {
      //print "Writing tag " . $row['tag'] . " for id " . $id . "<br>";
      $totalTagsWritten++;
      writeToFile( '<category domain="post_tag" nicename="'.makeScoopStyleNiceName($row['tag'], false).'">'.wxr_cdata($row['tag']).'</category>' );
    
    }
    // Free the resources associated with the result set
    // This is done automatically at the end of the script
    mysql_free_result($result);
}


function insertComments($story_id) {
  global $dbConnection1, $dbConnection2;
    // pull tags.  Scoop doesn't have categories, so all tags will be from 'uncategorized'
    $query = 'Select c.id, c.uid, u.nickname, c.commentip as ip, UNIX_TIMESTAMP(c.date) as date, c.subject, c.comment, c.pending, c.pid FROM comments c, users u ' .
             ' WHERE c.story_id='.$story_id.
             ' AND c.uid=u.uid';
    
    $result = mysql_query($query,$dbConnection1);
    if (!$result) {
        $message  = 'Invalid COMMENTS query: ' . mysql_error($dbConnection1) . "\r";
        $message .= 'Whole COMMENTS query: ' . $query;
        die($message);
      }
    
    // output the tags
    while ($row = mysql_fetch_assoc($result)) {
      
      //ex: 2009-01-07 17:30:08
      $commentDate_gmt = date("Y-m-d H:i:s", $row['date']);
      
      writeToFile( '<wp:comment>'.
      '   <wp:comment_id>'.$row['id'].'</wp:comment_id>'.
      '   <wp:comment_author>'.($row['id'] == -1 ? 'anonymous' : wxr_cdata(changeNickName($row['nickname']))).'</wp:comment_author>'.
      '   <wp:comment_author_email></wp:comment_author_email>'.
      '   <wp:comment_author_url></wp:comment_author_url>'.
      '   <wp:comment_author_IP>'.$row['ip'].'</wp:comment_author_IP>'.
      '   <wp:comment_date>'.$commentDate_gmt.'</wp:comment_date>'.
      '   <wp:comment_date_gmt>'.$commentDate_gmt.'</wp:comment_date_gmt>'.
      '   <wp:comment_date>'.$commentDate_gmt.'</wp:comment_date>'.
      '   <wp:comment_date_gmt>'.$commentDate_gmt.'</wp:comment_date_gmt>'.
      '   <wp:comment_content>'.wxr_cdata('<b>'.$row['subject'] . '</b><br/><br/>' .$row['comment']).'</wp:comment_content>'.
      '   <wp:comment_approved>'.($row['pending'] == 0 ? '1' : '0').'</wp:comment_approved>'.
			'   <wp:comment_type></wp:comment_type>');
              // Subquery to figure out parents
              $query2 = 'Select id FROM comments ' .
                        ' WHERE story_id='.$story_id.
                        ' AND cid='.$row['pid'];
              
              $parentComment = mysql_query($query2,$dbConnection2);
              if (!$parentComment) {
                $message  = 'Invalid Parent Comment query: ' . mysql_error($dbConnection2) . "\r";
                $message .= 'Whole Parent Comment query: ' . $query2;
                die($message);
              }
              
              // output the tags
              $foundParent = false;
              while ($parentCommentRow = mysql_fetch_assoc($parentComment)) {
                writeToFile( '   <wp:comment_parent>'.$parentCommentRow['id'].'</wp:comment_parent>');
                $foundParent=true;
                break;//there can be only one!  but just in case... 
              }
              if (!$foundParent) {
                writeToFile( '   <wp:comment_parent>0</wp:comment_parent>');
              }
              mysql_free_result($parentComment);
			writeToFile( '   <wp:comment_user_id>'.($row['uid'] == -1 ? '0' : $row['uid']).'</wp:comment_user_id>');
      writeToFile( '</wp:comment>');
    }
    // Free the resources associated with the result set
    // This is done automatically at the end of the script
    mysql_free_result($result);
}

/*
 * This is important... WordPress's import tool crapped out on me 
 * when it hit characters it couldn't handle... like control chars,
 * which took me a long time to figure out.. 
 */
function fixOldHtmlCodes($str) {
    // build patterns
    $patterns = array("/[&]rsquo[;]/", "/[&]ldquo[;]/", "/[&]rdquo[;]/", "/[&]mdash[;]/", "/&(?!(?i:\#((x([\dA-F]){1,5})|(104857[0-5]|10485[0-6]\d|1048[0-4]\d\d|104[0-7]\d{3}|10[0-3]\d{4}|0?\d{1,6}))|([A-Za-z\d.]{2,31}));)/" );
  $replacements = array("&#x2019;", "&#x201C;", "&#x201C;", "&#x2014;", "&amp;" ); // These are &amps; that are not part of an html code group
  
  array_push($patterns, "/[&][#]8220[;]/", "/[&][#]8221[;]/", "/[&][#]8217[;]/", "/[&][#]8212[;]/");
  array_push($replacements, "&#x201C;", "&#x201C;", "&#x2019;", "&#x2014;");

  return preg_replace($patterns, $replacements, $str);
}

function writeToFile($text, $writeOrAppend = 'a') {
  global $loopVar, $numOfWXRFilesToCreate;

  $myFile = "wordpress_import_".($loopVar + 1)."_of_".$numOfWXRFilesToCreate.".xml";
  $fh = fopen($myFile, $writeOrAppend) or die("can't open file");
  fwrite($fh, utf8_encode ( $text . "\n") );
  fclose($fh);
}

// From WordPress... not sure i need it.  I was having issues with CDATA and this is 
// EXACTLY (even the function name) of the code used in WordPress.  I added my own
// filtering to it, but the CDATA line is theirs.
function wxr_cdata( $str ) {
    $str = fixOldHtmlCodes($str); 
    $str = preg_replace('/[\x00-\x1F\x7F]/', '', $str);

		$str = utf8_encode( $str );

		// $str = ent2ncr(esc_html($str));
		$str = "<![CDATA[$str" . ( ( substr( $str, -1 ) == ']' ) ? ' ' : '' ) . ']]>';

		return $str;
	}
  
/* 
 * Change all authors to have a suffix so they
 * do not cause problems for people who want the same
 * name in the new system.  
 *
 * You can get rid of the suffix if you don't have a 
 * huge number of authors... you just have to change
 * each one's password, i think... i didn't do that.
 */
function changeNickName($name) {
  return $name . '_scoop';    
}

// Function for basic field validation (present and neither empty nor only white space
function IsNullOrEmptyString($question){
    return (!isset($question) || trim($question)==='');
}


  
?>
Share

Fixing authors when importing a split WXR file for WordPress

If you have a large WordPress Extended RSS import file, WordPress suggests that you split the file in order to create multiple smaller files.  If you do this, then each file will have the same authors.  If you import the second file, third, fourth, etc… then those authors will be seen as duplicates and dropped (and all imported posts will be credited to ‘admin’) unless you go through each author and select them in the “or assign this post to existing author” combo boxes.   These combo boxes become available after you select a file and press “Import.”

If you are importing multiple files because a single one is too big, there is a chance you could have way too many authors to go through each one and do this.  This post describes a hack method of populating all of these combo boxes using jQuery and the browsers development tools.

I had to import 6 WXR files that i created, each one around 80Megs, and a total of almost 500 authors.

1) Get jQueryify here and add the bookmarklet per the instructions.

2) When you get to the “assign authors” page, right-click on the browser page and select “Inspect Element” (Chrome), or “Inspect with Firebug” in Firefox (install firebug if you haven’t already). In IE press F12 to start the Dev Tools.

3) Press the jQuerify bookmarklet button that you installed from step 1. This will turn on jQuery for the page.

4) Go to the console tab, enable the console if you need to, paste this code in and press RUN:

jQuery.noConflict(); //Disable jQuery $ just incase.

jQuery('#authors li').each(function() {
  // Get the author login text
  var username = jQuery(this).find('strong').html();
  var author_login = jQuery.trim((username.split('('))[0]);

  //Figure out which option this author is in the drop down.
  var selectOptionval = -1;
  jQuery(this).find('select option').each(function(){
    if (jQuery(this).html() === author_login) {
      selectOptionval = jQuery(this).val();
      return false;//quit .each() early.
    }
  });

  // Set the combo box to this author's option key.
  jQuery(this).find('select').val(selectOptionval);

  // For test...
  //console.log(author_login + ": " + selectOptionval);
});
Share

jQuery ready function with holdReady

I used this the other day in GWT, prior to backing it out and placing style classes in better places so i could use straight CSS targeting.  But, I thought it could be useful and it uses the new jQuery.holdReady function so here it is.

My goal was this: In order to get the correct horizontal scrollbar in a GWT app, I had to target the div right ABOVE the root div placed in my Root View (which contains the structural layout of my page). I used jQuery to target that div, but they went back and placed a styleclass in the RootLayoutPanel widget instead so that i could target this more cleanly (within GWT) with CSS’s :nth-of-type / :nth-child selector.  I found the exact DIV that i needed to target by placing overflow-x: auto !important on all the parent divs until it worked ‘correctly.’

The ‘pageScrolling’ style class is defined with only overflow-x: auto !important

/* Startup functions */

/*
* First, place a hold on the document ready function... GWT loads more stuff
* after the DOM loads, and we need to add a little extra delay to account for that.
*
* By using document ready and hold ready together, i am able to make sure that the
* startup function runs after the page is completely ready.
*/
jQuery.holdReady(true);

// Set up the on-DOM-ready function
jQuery(function($) {
  // Find the DIV that should handle scrolling for the "body" and mark it
  $('.myRootPanel').parent().addClass('pageScrolling');
});

/*
* GWT needs to create its structure, which happens after DOM loads??
* Therefore, hold the document READY function until an expected element
* is detected
*
* (ps.. This holds ALL jQuery ready functions anywhere the system... if there are any)
*/
var waitForPageToLoad = function() {
  if (jQuery('.myRootPanel').size() &gt; 0) {
    // it exists now.. trigger the ready function
    jQuery.holdReady(false);
  } else {
    // Doesn't exist yet.. wait some randomly selected time period.
    setTimeout( waitForPageToLoad, 200 );
  }
};

/*
* Trigger the timeout function to wait for the page to REALLY load,
* and then release the document READY hold.
*/
waitForPageToLoad();
Share

Dynamically remove blank cells from a column in Excel

This is another Xcelsius issue i had — but the solution is usable in any excel spreadsheet.

I wanted to create a fake drop down (custom image w/ little arrow like most web pages have now + push button + list box) that i could set up with dynamic options — dynamic meaning different options based on different states of the dashboard, not adding or removing new options at runtime.  The problem was that i needed to link to a range in the Excel spreadsheet that gave me the currently available options and did not have blank spaces between them.  I hunted on the internet and found an example, but it used functions that are not available in the Xcelsius-enabled list of Excel functions.

So I edited it to be Xcelsius-friendly and here it is (click on this pic to see a larger version):

Table layout for dynamically removing blank cells
Basic table layout for setup
  • Column 1 (H): a row number that is copied across to column 4 when the option text is marked as “available” in column 3.  these are hardcoded numbers.  Xcelsius doesn’t support the ROW function.
  • Column 2 (I): The table of data.  In my case it doesn’t have spaces, but effectively does when i have certain options marked as “unavailable”.
  • Column 3 (J): A formula that is unique for each row and determines if THAT row should be available.  This checks the status of different things in the Xcelsius file (e.g., the dynamic visibility of other components in order to determine what tab on on, etc) and determines when this option should be available in my faked-out combo box implementation.  If you have data that has spaces in it that you are removing, this row can check for that or (even better) just combined with column 4.
  • Column 4 (K): A filter.  If this row is “available”, then this cell is set equal to the row number in column 1.
  • Column 5 (L): blank… i left it there for spacing, or maybe for adding something later… i don’t know.  I can’t remember.  but it is blank.
  • Column 6 (M): The magic happens here… Keep reading

The function in Column 6 is this, the column letters are defined in the column definitions above (i.e., column 1 is H) and the first row of that table was at row 30:

=IF(ISERROR(SMALL($K$30:$K$49,H30)),"",INDEX($I$30:$I$49,SMALL($K$30:$K$49,H30)))

To break this down:

  1. ISERROR(SMALL($K$30:$K$49,H30)),””…: The first part here checks if there is a next smallest value in column 4.  if there is, it continues to the INDEX function, otherwise it leaves the cell in column 6 empty.  Column K is the sparsely populated filter column (aka column 4), and the “n” in the “nth next smallest value” comes from column H (the rownum).
  2. INDEX($I$30:$I$49,SMALL($K$30:$K$49,H30)):  Since we know an “nth next smallest value” exists in the filter column, this pulls the “option text” in column 2 (i.e., the table data) corresponding to that row number.  the row number starts at 1 for the first row, so it is equal to the “index” of that row when using the INDEX function.
Here is a screen shot to help you line up the formula to the columns.  If you setup the first cell in column 6 with the same “$” that i use for static cells, then a copy-down will fill in that whole column correctly:
Options table with formula 1st row
Showing the super awesome formula in the last column

It is actually pretty simple and useful.  I simplified it as i wrote it here because i found a function that I was using that was redundant, but in case it solved some other issue that i don’t have and can’t think of, here is it:

=IF(ISERROR(SMALL($K$30:$K$49,H30)),"",INDEX($I$30:$I$49,MATCH(SMALL($K$30:$K$49,H30),$K$30:$K$49,0)))

.   The MATCH at the end there is redundant.  In my example, the relative index is also equal to the “nth next smallest value” in the filter column — so i’m doing a MATCH using the SMALL function to get the same number the SMALL function already returns.  It is useful to write these posts and fix my own code!

Share

BI: Using combo boxes to get things done in Xcelsius

I was on a project recently working with dashboarding in SAP Business Objects and I found a couple things that helped me out so i wanted to put them up here.  I’m heading back to the land of web development so i’ll document these things before I forget them.

This post is one about using combo boxes in Xcelsius to actually do some things.

Xcelsius is a dashboarding tool — not a UI tool.  I’m used to UI tools which let me use callbacks and events,etc.  Xcelsius is very simple and only lets you do simple actions. Everything is based on performing some action (e.g., press a button), change some value in the attached Excel spreadsheet, and (typically) hide or show some other things based on that value you just set.  So you see here that for a given action, you can only (easily) perform one data movement (called data insertion).  There are other components, like tables, that will sometimes let you move more data when you click or hover (your only two event options… and they cannot be used at the same time on the same component), but most let you do a single thing.

With combo boxes, you can do more.  I do not know if this is an intended feature or not, but you can set up hidden combo boxes to watch for that “single thing” that most components do, and then perform a lot of data insertions.  This, along with some nice excel functions, allows you to cross into the realm of making the dashboard act like a more complex UI.  I didn’t find any good write-ups so here is mine.

Anyway, here’s the idea:

A couple things to know to start…

  1. A combo box ONLY watches for the trigger and performs the data insertion when it is “visible”. This means that if it is hidden because of its Dynamic Visibility setting on the Behavior tab, OR if it is on a canvas that is not current visible, etc, then it WILL NOT RUN. This is GOOD. it allows additional control to help avoid getting into data insertion loops or having timing issues.
  2. While a combo box only runs while it is “Dynamically Visible,” any functions (like IF statements) that you put in the spreadsheet cells continue to run all the time.
  3. If you are using the combo box for data insertion only and do not want the user to interact with it, you need to hide it so it is not “physically” visible when it is set to be “dynamically visible” (ie, put it behind another component so it runs but isn’t seen.) You need to move it up or down in the Object Browser to get it to the right place for it to (a) run when you want it, and (b) not be seen if you don’t intend it to be.
  4. I have noticed that the combo boxes executed in the order they are placed from TOP TO BOTTOM in the object browser.

The settings you need to use it for Data Insertion…

When you add the combo box, it is very simple to set up. This is the way i use it for data insertion. If you know a new way, edit this document!

Behavior tab

  1. Set the TYPE to “Label” and the ITEM to the cell you want to WATCH.  This means the cell you are watching is your trigger cell. When this cell becomes equal to a value that is in your list of “labels” (that you will define on the General tab), then the data insertion will occur.

General tab

  1. Set the LABELS cell or range to values that are valid for kicking off this data insertion. These would be what are shown to the user in the combo box if it was visible.
  2. Set the “Clear destination when no selected item” if you want that… make sure to consider if you have other people (“components are people, too”) copying data to this destination under different circumstances.
  3. Set the radio button at the bottom to “Data change and interaction” — “Interaction only” doesn’t make sense for a hidden component.
  4. Finally, set up the data insertions!
    1. If you select Filtered Rows, then you can only have one data insertion. I’m not sure if the labels and the filtered rows have to match up or not for this. If you want to move multiple rows, like a whole table, then you need to use this one. I could not get a large block of data to move when using data insertion types other than this.
    2. Any of the other insertion types can have multiple data insertions in the queue. This is great… off of one trigger you can rearrange all kinds of stuff, and the source and destination do not have to have any relation at all to the labels or trigger cell. Examples are:
      1. Value: This will move ONLY a single value from one cell to another.

        Important Note: Per the Xcelsius 2008 user guide ( and my experience ), it is REQUIRED that the VALUE data insertion SOURCE range is equal in size to the LABEL range. This requirement can lead to some columns in the dashboard spreadsheet basically being hardcoded, repeated identical value columns if that is what you need.

      2. Row: This will move a single row… not multiple.. (at this is what i saw when testing).
      3. Column: This is like ‘row’ except it moves only a single column

Additional Note:

Avoiding selection memory in the combo box: You may want to set the Item field to a cell. This is on the Behavior/Common tab of the combo box component. If you are using the combo box behind the scenes, or if you have it visible with only one selectable item, then once a list item is selected you may not be able to select it again until a different item is selected (or not at all if you only have one item in the list). A solution for this is to set the Item field equal to the Insert Selected Item field, and then make sure that you clear that cell after a selection was made and the action performed (ie, make the last data insertion in the combo box move a blank value to that cell). By keeping that cell cleared after a selection is made, and by having these two fields equal, there will never be a “pre-selected” item in the list.

So that’s it.  You can use combo boxes to do more in Xcelsius.  Let me know if this helps you out!

I have two more write-ups that i will do when i have a chance:  One is an useful excel function that lets you take a column of blank and populated cells and compress them into a single column without spaces, and the other shows how to create a counter using the History component.

Share

Dropbox install on Fedora 14

I started using Dropbox and couldn’t get it installed on my 64 bit fedora 14 machine.  I found the solution and am just reposting here to hopefully guide someone to it quicker than I was able to find it.

Here is the dropbox forum page with the information and description of the issue (the answer that worked for me is the last post, by P K.)

Here is the solution in case you have dropbox installed, it is not working, and you don’t want to go through the forum post:

execstack -c ~/.dropbox-dist/_ctypes.so

If you haven’t installed it yet, here’s how I did it:

  1. I downloaded the fedora rpm file from the dropbox site.  Get it here.
  2. I installed it using yum localinstall <packagename>.rpm.  This gave me “missing public key” error.  The solution here was to edit (as superuser) the /etc/yum.conf file and temporarily disable gpgcheck (ie, change gpgcheck=1 to gpgcheck=0).  Edit the file with something like sudo gedit /etc/yum.conf
  3. You can skip this step, unless you want to see the problem.   Once installed, run dropbox — either from the command line or the from the Application Launcher, under the Internet folder.  This says you have to run a daemon, press OK.  It says it is downloading and unpacking… and never finishes.
  4. Don’t forget to go back into /etc/yum.conf and reset gpgcheck=1
  5. In the terminal, type execstack -c ~/.dropbox-dist/_ctypes.so <enter>
  6. Now, run dropbox from either from the command line or the from the Application Launcher, under the Internet folder.
That’s it.  It is working for me.  I was using the Dolphin file explorer but dropbox installs into the Nautilus file explorer… so i’m using that now.  Dolphin crashed often for me anyway, so i was never really attached to it.
Share