ng-admin + JAX-RS: 400 Bad Request on DELETE

I’m tried of building admin UIs and I’m trying out ng-admin. It’s pretty straightforward to setup given the guides and demos.

List, create, updates were fine until I got to the DELETE method. The server was throwing 400 Bad Requests and upon Chrome network inspection I discover that ng-admin was sending a JSON body in the request. I don’t really care who is “following the standard” as long they work together (think browsers and jquery), so I’m fine to fix either side to either the client not send the body, or the server accepting the non-empty body.

ng-admin uses Restangular under the hood to make REST requests. Restangular did have this FAQ about DELETEs with body(s).

A little refactoring and presto! DELETE now works.


app.config(['RestangularProvider', function(RestangularProvider) {
  RestangularProvider.setRequestInterceptor(function(elem, operation) {
    return (operation === "remove") ? undefined : elem;
  });
}]);

multiple definition of `R_running_as_main_program`

I inherited a C++ application that was using rcpp to embed R. After R was upgraded to 3.2, the make was failing miserably with the error:

file1.o:(.bss+0x0): multiple definition of `R_running_as_main_program'
file2.o:(.bss+0x0): first defined here

This was found to be caused by https://bugs.r-project.org/bugzilla3/show_bug.cgi?id=15899

To overcome this, edit /usr/share/R/include/Rinterface.h and search for this line:

int R_running_as_main_program;

Add the “extern” keyword at the start and save it:

extern int R_running_as_main_program;

After that there should be no problem building the program.

Wrong encoding when JSP includes HTML

We had a standard JSP header with logo and menu, and we used it to include a few static HTML pages (e.g. privacy policy and about us). We had multiple versions of the static content in different languages, and the non-English ones shows up in gibberish, although the menu shows up fine.

We already specified the encoding in our enclosing JSP page.

<%@ page pageEncoding="UTF-8"%>

as well as specified the charset in the meta header

A common “workaround” is to convert the html to JSP to specify the encoding. However being lazy developers, we want a better solution. Eventually we found the hint to set the encoding in web.xml. We tweaked the extension from .jsp to .html and it works!



  
    *.html
    UTF-8
  

Lenovo T440p keyboard

I was just issued a Lenovo ThinkPad T440p, not that I get to choose the model. Immediately I got down to customizing the quirks (to me).

1. Fn/Ctrl key swap
Ctrl-C Ctrl-V Ctrl-Z Ctrl-W all don’t work. Because in the Ctrl’s position is a Fn. This link explains why (so we could find the Fn key for ThinkLight in the dark), but I didn’t really need the ThinkLight so I swapped it anyway.

Go to BIOS > Keyboard/Mouse > Fn/Ctrl Swap > Enabled.

2. F1-F12 lock
F1-F12 could only be accessed with the Fn key. so F2 to rename became Fn-F2, and F5 to refresh became Fn-F5.

Press Fn-Esc to lock the Function keys.

3. Trackpad Scroll
I use a MBA at home so the two-finger scroll was reversed.

To keep myself sane, go to Control Panel > Mouse > Change Mouse Settings > ThinkPad > Advanced > Scroll > Two-Finger Scrolling > Check Switch Direction.

4. Lenovo Message Center Plus
The big red icon on the taskbar was an eyesore to me.

Right click on the taskbar > Toolbars > Uncheck Lenovo Solution Center.

Glassfish timezone different from OS (Ubuntu)

We added a new Ubuntu server to deploy one of our new feature to isolate it from the core modules. The new module did not work, and we narrowed it to System.currentTimeMillis() returning a time in the future.

NTP was active; we ran the linux “date” command and it was showing the correct date. When we checked Glassfish’s JVM report, it showed that the user.timezone was different, and the timezone difference coincides with the time differences we observed. The straightforward answer: set -Duser.timezone in the JVM options and we are good to go.

But wait, why our original servers didn’t have this issue? We checked the original server, and there was no such JVM option setting. After some tracing it turns out that /etc/timezone of the new server was incorrect, and that influenced the timezone Glassfish used. Finally we matched the two server settings and reverted the user.timezone JVM option.

mysql ibdata keeps growing

Didn’t manage to collect complete information for this post, so I’ll just write whatever I have.

It all began when our development database server ran out of disk space and crashed. We mounted a temporary virtual hard disk and moved the database there, and all was well for a while. With df and du we narrowed our culprit to mysql’s ibdata, which was growing so fast that we will run out of disk space again soon enough. Public information tells us ibdata is supposed to always grow, but we do not expect our dev ibdata to grow at this rate. After multiple searches this blog finally closes us in.

SHOW ENGINE INNODB STATUS

With the command, the innodb history list length was a very large number (>1mil) and keeps going up. A check with our test and production databases show that the length goes at most to a few hundred and drops back down — a significant difference. Restarting the database doesn’t help.

SELECT * FROM INFORMATION_SCHEMA.INNODB_TRX

This showed us we had 16 XA transactions, that were started 2 days ago, but never committed or rolled back. They are not locked, so their trx_mysql_thread_id is 0. We immediately linked the causes together. Stuck XA transactions -> history list growing.

XA RECOVER

According to mysql docs, this command can rollback the XA transactions. The user comment was especially helpful on how to reproduce the xid, reproduced verbatim here:


To rollback the transaction, first get its xid:

mysql> xa recover;

+----------+--------------+--------------+------------------------------------------------------------+
| formatID | gtrid_length | bqual_length | data                                                       |
+----------+--------------+--------------+------------------------------------------------------------+
|   131075 |           30 |           28 | 1-a00640d:c09d:4ac454ef:b284c0a00640d:c09d:4ac454ef:b284c2 |
+----------+--------------+--------------+------------------------------------------------------------+
1 row in set (2.13 sec)

The xid is present in this output, but you have to perform a little string manipulation to get it. The format of a xid is: gtrid,bqual,formatID. The column 'data' contains a concatenation of 'gtrid' and 'bqual'. The columns 'gtrid_length' and 'bqual_length' specify how many bytes each of these values uses; use them to split apart 'data'. In this example, the result is:

mysql> xa rollback '1-a00640d:c09d:4ac454ef:b284c0','a00640d:c09d:4ac454ef:b284c2',131075;

ERROR 1402 (XA100): XA_RBROLLBACK: Transaction branch was rolled back

The tricky part here was that, my data had binary characters, which I couldn’t directly copy and paste in the MySQL Workbench. I couldn’t bear to write a program to read the value and write it back either, so I was poking around for solutions on that. From the same mysql doc page,

gtrid and bqual must be string literals, each up to 64 bytes (not characters) long. gtrid and bqual can be specified in several ways. You can use a quoted string ('ab'), hex string (0x6162, X'ab'), or bit value (b'nnnn').

Good, I could write the xid in hex. So I right-clicked the data column, and “Open Value in Viewer”. In the binary tab I copied down the hex values and reconstructed the xid as described by the helpful comment.

XA ROLLBACK X'7e3ae860eb21de21b84d392cb03bf8363b41482b9b1207f6e6823355012e91858c',X'526801e7500b06fd05a2f5882d20be19982a46aed4b6c26dc63887',4264851;

Viola, one by one the transactions were gone. Once the last one was rolled back, the history list started to decrease and behave in a similar pattern as our other databases, and the ibdata stopped growing at the crazy rate. The one last part I haven’t figured out is: how do I copy the hex values from MySQL Workbench, or how do I show the XA RECOVER data column in hex?

Firefox marquee restarts prematurely

I was trying to fix a HTML marquee bug today, where only on Firefox the marquee restarts itself before the whole message scrolls to the end. After stripping down the page the cause appears to be Firefox observing the length of the original text (which was just a single space character).

The HTML code below reproduces it. Click on “Change Marquee Text” to see the difference between the two marquees.




At first


Later, the marquee text became longer but Firefox does not recognize it.

Not worth to further investigate here, I just dynamically re-rendered a container div for this marquee.

double HTTP requests

We first discovered that our webapp was receiving duplicate HTTP AJAX requests from clients that results in a database insert. Fortunately jQuery had a nocache timestamp as part of the AJAX request so we could recognize it as a duplicate and reject the 2nd request.

As we tried to narrow down the cause we found that it happens on both GET/POST, as well as on a variety of browsers and network providers. After days of trying to reproducing the behavior we resorted to using iMacro to do browser automation and finally manage to reproduce it occasionally. What was surprising was that we were receiving the response of the 2nd response, which was the rejection message! (while the database insert was successful) It was absolutely confusing and we set up the automation again with Wireshark.

The packet analysis confirmed that the client browser was sending the duplicate request. However we also scanned Firebug and confirmed that only a single AJAX call was made! It didn’t make any sense to any of us until I noticed a pattern in the Wireshark logs that the 1st request made was always terminated without a response (either a TCP Connection Reset or a proper TCP Connection Close initiated by the server), and the 2nd request would be made on another kept-alive TCP connection. Following that symptom with Google I chanced upon a blog that highlighted HTTP 1.1 RFC 8.2.4.

If an HTTP/1.1 client sends a request which includes a request body, but which does not include an Expect request-header field with the “100-continue” expectation, and if the client is not directly connected to an HTTP/1.1 origin server, and if the client sees the connection close before receiving any status from the server, the client SHOULD retry the request.

Could this be it? We mocked up a HTTP server using raw Java ServerSockets by intentionally closing the first connection and allowing the second. We logged the requests on the server-side and voila! The mock server showed that it received two HTTP requests, Firebug showed that it sent one only, and Wireshark showed that the browser sent two.

Essentially, we just proved that the browsers adhered to HTTP 1.1 RFC 8.2.4…….