Posts tagged “c”
I like Ruby. Ruby the language anyway. I have started to have violent
reactions to PHP's lack of closures. However, Ruby has one deficiency that
makes me almost as mad as PHP's lack of closures. Assume for this example
that the file foo.rb looks like this:
@foo = "I'm getting ready to abuse the instance variable syntax."
As is, Ruby lets me do the following, which is functionally desirable
but makes the good programmer part of me twitch:
class Foo
def initialize(path)
self.instance_eval(File.read(path))
end
end
f = Foo.new('load_me.rb')
f.instance_variables # => ['@foo']
Replacing the bold line with the following is what I'm after:
load_in_scope path
In my dream, load_in_scope is a function defined in the C
code that compiles a file using the Bison parser to read through the file.
This saves a whole round trip to and from memory, making the program more
efficient and me less twitchy.
I'm having major trouble getting the Ruby interpreter to do this.
Starting with a trunk checkout and working backwards from the
Kernel#load method, I've been able to segfault at least
two-dozen different ways. I think there are only two changes that are
necessary. First, after rb_load_file returns a NODE
*, rb_iseq_new must turn that into an instruction
sequence with a type other than ISEQ_TYPE_TOP (perhaps
ISEQ_TYPE_CLASS or ISEQ_TYPE_EVAL?). Second,
the instruction sequence can't be passed to rb_iseq_eval
because that calls vm_set_top_stack, so
load_in_scope must call vm_set_eval_stack and
vm_exec itself.
Am I insane?
Comments (0)
mtrace(1) and mtrace(3) are my new bicycle. The
manual for debugging tool design should read SEE ALSO mtrace(1),
mtrace(3). One header file, one environment variable and one on/off
switch.
The catch to this free lunch is that it doesn't completely work with C++.
If run over a C program, it can print the file/line that allocated a leaked
block. If run over a C++ program, it can't do anything but count
blocks that are allocated but not freed. This is still gun-in-a-knife-fight
useful. Saying it's a C++ issue is a bit of a lie: because the
new and delete operators use malloc
internally, allocations using these operators are reported as coming from
libstdc++. If you use malloc in C++, it works fine. Because
the actual call to malloc or free is within
new or delete, the file/line notation is at best
useless and at worst an unadorned memory address.
I found it beneficial to create a single-threaded "test script" version
of my program for mtrace-ing. With a bunch of threads thrashing about,
there was too much noise from freeing memory that had been allocated in
other threads for the output to be very useful. Fortunately, removing
threads was (for me) a functional and expedient solution.
The bug turned out to be difficult to find but embarrassing, so I'm
not going to tell you what it was.
Comments (4)
I've been meaning to write this post for some time to document how I avoid the giant-if-statement-with-strcmp() problempattern when writing C programs.
The solution to every problem in C is to use pointers. Keeping with that lemma, it's possible, beneficial and even fun to make pointers to functions do your bidding without the pile of if or switch statements typical of most student programming projects. It's really close enough that I started calling it functional programming years ago even though this is still a far cry from Lisp/Python/JavaScript/whatever. It doesn't matter because C is still awesome.
Use the STL, Luke
I'm cheating a bit when I say functional programming in C. The use case I'm going to present is a C++ sample but there's no reason why you can't hash your own table in straight C99. I just don't care to myself. The example uses the STL map class to map strings to functions, making it trivially easy to build your own shell-like app that takes some argument and dispatches a function to make it so. The map provides the nice dispatch-y interface but there's no reason you couldn't do something like this in plain C:
void foo(int i) {
printf("You called foo(%d)!\n", i);
}
void bar(void(*)(int) f) {
(*f)(2);
}
Anyone fluent in a scripting language that isn't PHP will likely understand exactly what's going on here. foo is a simple function that does nothing interesting. bar is a function that takes as its argument a pointer to a function. That function can't return anything and must take a single integer argument. Many will call foul here saying that the strong type restrictions here make this sort of "functional programming" rather useless. I say nay and cite curvr.cc as my example. In any language, a function lacking any expectations for its arguments will be rather tedious. C doesn't give you this as a way to shoot yourself in the foot (there are so many other ways, however).
Function pointer declaration
The simplest way I know to explain the difficult syntax of function pointer declaration is an analogy to something easy like a character and a pointer to a character.
char c : char * p :: void foo(int i) : void (*)(int)
The first half declares a character, c and a pointer, p that can point to c. Similarly, the second half declares a void function foo that takes an integer argument and a pointer to a void function that takes an integer argument.
Now for the useful part
Using this pattern for creating function pointers we can now define a few functions and map them in a useful way. This is again taken from my curvr.cc file (more info about curvr).
// First a few function prototypes to map
int curve(Magick::Image & img);
int bigcurve(Magick::Image & img);
int anticurve(Magick::Image & img);
// Then the map part
int main(int argc, char * * argv) {
// Magically figure out a string called process
// Setup the map using the address operator (&)
std::map<std::string, int (*)(Magick::Image &)> processes;
processes["curve"] = &curve;
processes["bigcurve"] = &bigcurve;
processes["anticurve"] = &anticurve;
// Call a function from the map
(*processes[process])(img);
}
Your code will be shorter now. Go forth and hack.
Comments (3)
Just like the career of the average software engineer, Curvr has now grown up just a bit and started reusing other people's code.
Dopplr integration in curvrmail
Geotagging is awesome but a bit of a pain to automate with any kind of accuracy. I'm making a compromise here by automatically tagging things using my location according to
Dopplr. I had this working with my original PHP
curvrmail script but have since switched to using
Flickr::Upload::Dopplr just to practice some Perl.
AutoManual image rotation
Since my N73 isn't friendly enough to rotate pictures for me, I've added a way to specify rotation in the email you send to Curvr. Prefixing with "L " or "l " will rotate 90 degrees counterclockwise and prefixing with "R " or "r " will rotate 90 degrees clockwise. Note in both cases that it is a letter followed by a space followed by the normal title/tags syntax.
It is a bit awkward to think about in terms of left and right, since I could either be talking about the direction you rotated the camera or the direction the photo should be rotated (which itself could be rather ambiguous, so we'll say that it is relative to the top). So here's the official word: the rotate syntax wants to know which direction you rotated the camera to take the picture.
curvrconf for getting API tokens
Getting tokens for the Flickr and Dopplr API, while not something you need to do often, is quite a drag. Enter
curvrconf, the copy-and-paste simple way to get your tokens out. All it needs is to be setup with the same API key as
curvrmail and it'll guide you through grabbing tokens. For the Flickr API, setup your key for web-based auth and give it a garbage URL to return to (I use http://localhost:81/).
As usual, the API key is up to you. I accidentally committed my keys/secrets/tokens to Subversion but have expired all of them so don't even bother trying to steal my accounts.
Anti-blue C code
The main
curvr program has for a long time introduced some strange blue artifacts into my photos. I figured this was due to some underflow or overflow issues with my color maps. I finally took the time to do some serious experimentation and it turns out that if I just favor a stronger red channel in these edge cases, the artifacts seem to go away. Strange and beautiful indeed. I can't fully explain the phenomenon but believe it's somehow related to my using the red channel as my indicator of color value (as many photographers do when converting to black & white).
So there it is, curvr-0.2.
Er, here it is: http://svn.rcrowley.org/svn/curvr/tags/curvr-0.2
Comments (0)
Manifest (verb): To embed XML describing libraries which are expected to be available at runtime, done in the course of developing software using Microsoft Visual Studio.
The wisdom of the collective T-Rex (#xulrunner on irc.mozilla.org) says that statically linking against the Microsoft C Runtime (msvcr80.dll and friends) is the way to go. However I've run into a case where I cannot do this (deciding whether this is due to technical limitations or lack of competence is for readers to help me decide).
The image processing component of the Flickr Uploadr links against GraphicsMagick and Exiv2, libraries that do image processing and EXIF/IPTC metadata processing, respectively. Without them, it doesn't fly. Problem is, they both link the C Runtime themselves by using std::string and such. Both of these libraries will build fine given the /MT flag, which instructs them to statically link the C Runtime. When they themselves are statically linked into the XPCOM component later, all hell breaks loose. It seems that the static linker can't resolve the duplicate definitions caused by overzealous static linking.
The solution, as it stands, is simply to include the C Runtime and its manifest in the distribution. Placing the following files in Uploadr's components/ directory has allowed life to go on for the moment.
msvcr80.dllmsvcp80.dllmsvcm80.dllMicrosoft.VC80.CRT.manifest
Lastly, make sure your XPCOM project in Visual Studio is set to embed its manifest from Project Properties » Manifest Tool » Input and Output » Embed Manifest.
A more optimal solution might be to combine the Visual Studio projects for GraphicsMagick (and therefore libjpeg, etc), Exiv2 and the XPCOM component into one large tree that Visual Studio might be able to better manage. Has anyone out there done such a thing with Microsoft's static linker (even if not with XULRunner)?
Comments (3)
This post is about two things. Most proudly it is about me finally taking the time to setup all of the necessary gears and levers for my phone to post processed photos through to Flickr. Bonus-ly, it is about rolling what seems like your own email server from a few config files and Yahoo! Mail (or GMail, if your twisted brain is so inclined).
First, the fun stuff. I posted in November about curvr, my automated and assumption-filled command-line Photoshop. Refresher course here: http://rcrowley.org/2007/11/08/introducing-curvr/. The day after I wrote curvr I started to experience what can only be described as "crunch time" on Flickr Uploadr, so a quick shell script called curvall was born so I could at least use curvr after Bluetoothing photos from my phone.
Almost two months later and this is just silly. A different kind of crunch time, if you will. From a distance the solution isn't nearly as ugly as the syntax of .procmailrc but I won't be mean. I setup an MX record for my domain to send mail to the box in my apartment. From there, postfix rejects the riff-raff and passes the good stuff on to procmail which either works photos over with curvr or forwards to Yahoo! Mail for my consumption.
Postfix, there's more to life than UNIX accounts
We need to teach postfix what email addresses to accept.
/etc/postfix/virtual:
r@rcrowley.org rcrowley@localhost
SPECIAL_CURVRMAIL_ADDRESS@rcrowley.org rcrowley@localhost
/etc/postfix/main.cf (
available in SVN):
# See /usr/share/postfix/main.cf.dist for a commented, more complete version
# Debian specific: Specifying a file name will cause the first
# line of that file to be used as the name. The Debian default
# is /etc/mailname.
#myorigin = /etc/mailname
smtpd_banner = $myhostname ESMTP $mail_name (Ubuntu)
biff = no
# appending .domain is the MUA's job.
append_dot_mydomain = no
# Uncomment the next line to generate "delayed mail" warnings
#delay_warning_time = 4h
# TLS parameters
smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
smtpd_use_tls=yes
smtpd_tls_session_cache_database = btree:${queue_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${queue_directory}/smtp_scache
# See /usr/share/doc/postfix/TLS_README.gz in the postfix-doc package for
# information on enabling SSL in the smtp client.
myhostname = banzai
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = /etc/mailname
mydestination = banzai, localhost.localdomain, localhost
relayhost =
mynetworks = 127.0.0.0/8
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = all
# Nasty homegrown config
virtual_alias_domains = rcrowley.org
virtual_alias_maps = hash:/etc/postfix/virtual
mailbox_command = /usr/bin/procmail
It really is important not to put the same domain in
mydestination and
virtual_alias_domains. Fecal matter and oscillating device and all that. Then run these commands:
$ sudo postmap /etc/postfix/virtual
$ sudo postfix reload
Procmail, your syntactic colon is full of shit
If photo, Flickr it. If not, throw it out into the cold.
~/.procmailrc (also available in SVN):
:0 bfW
* ^TO_<SECRET_CURVRMAIL_ADDRESS>@rcrowley.org
| /home/rcrowley/bin/curvrmail
:0 E :
! <normal_forwarding_destination>@yahoo.com
Now, Stephen R. van den Berg willing, email to your special address will be sent through
curvr and onto Flickr (assuming you updated the
curvrmail script with your secret Flickr email address) and all of your normal email will be forwarded somewhere else.
Yahoo-oooo!
I admit it, I like Yahoo! Mail. So I have my
.procmailrc forwarding to Yahoo! Mail. Over at Yahoo! Mail I have a default mailbox setup as the-first-letter-of-my-first-name@rcrowley.org so that I can also send mail as my better email address.
Anyway, it's all in SVN: http://svn.rcrowley.org/svn/curvr/
I'm interested to know how similar this all is to Aaron's filtr workflow.
Comments (5)
Why can't XULRunner do this already?
Without really saying what "this" is, Mozilla wants XULRunner to be the standard play for cross-platform GUI applications and yet it has a bit of trouble interacting with 3rd party libraries. Most mature libraries (say, GraphicsMagick) still use
char * or
std::string to represent file paths. To interact with them, then, you have to be able to get to a
char * representation of files you're trying to open.
Windows and Mac character encodings 101
As is becoming tradition, Windows makes my life difficult by using
UTF-16 as its Unicode of choice. UTF-16 uses 2 bytes (usually) to represent each character, meaning it isn't binary compatible with ASCII, which uses 1 byte per character. These 2-byte characters are stored in
wchar_ts which are twice as wide as
chars. It isn't possible to pass a
wchar_t * through normal C libraries that expect strings as
char *.
Mac OS X uses UTF-8 internally and is my hero. UTF-8 is used internally to represent strings and has the advantage of using single bytes (that's char for C programmers playing along) as its base unit. While a single character in UTF-8 can span up to 8 bytes, the in-memory representation of these characters will pass muster with a C compiler. When the UTF-8 bytes are passed through some C library they can arrive intact to the fopen system call, which understands the UTF-8 bytes and acts like you'd expect.
There's an easy way?
There could be. Using the
@mozilla.org/file/local;1 it might be possible to get the path as an
nsACString, Mozilla's portable representation of single-byte ASCII strings. But, the reference warns that the native path available from nsILocalFile is not for passing to C libraries and isn't guaranteed to be correct. So it's up to me.
No pain, no gain
Bear with me, this gets ugly. The solution that lets me take paths from JavaScript-land to C-land, open Unicode paths with a normal
char * and do so on Windows and Mac OS X is stupefying (and stupid). The iinitial path is stored as a JavaScript string, so the boundary between JavaScript and C will be an
nsAString, which represents strings as UTF-16.
The string takes quite a journey here, so let's cover the easy side first. Macs understand UTF-8, and I'm making a transformation to UTF-8 in JavaScript. This means that the only transform needed in C is a type change, not an encoding change. The UTF-8 string enters C-land as wide characters and is simply cast character-by-character down to an array of bytes.
Windows is quite a bit more work, since a path in Windows that needs Unicode characters can't easily be represented as a char *. Enter the Windows API function GetShortPathName. This can convert any path to an old-school Windows path that fits in ASCII and uses that familiar "8.3" naming convention. I got the Mac half working first, so the Windows implementation is actually more complicated than it needs to be -- more on that later. Without further ado:
string * conv_path(const nsAString & fake) {
// Fun with Windows paths
#ifdef XP_WIN
// UTF-16 but really UTF-8 nsAString to really UTF-8 nsCString
nsCString utf8 = NS_LossyConvertUTF16toASCII(fake);
// UTF-8 nsCString to UTF-16 nsEmbedString
nsEmbedString & utf16 = NS_ConvertUTF8toUTF16(utf8);
// UTF-16 nsEmbedString to wchar_t[]
wchar_t * w_arr = new wchar_t[utf16.Length() + 1];
if (0 == w_arr) return 0;
wchar_t * w_arr_p = w_arr;
PRUnichar * w_start = (PRUnichar *)utf16.BeginReading();
const PRUnichar * w_end = (PRUnichar *)utf16.EndReading();
while (w_start != w_end) {
*w_arr_p++ = (wchar_t)*w_start++;
}
*w_arr_p = 0;
// GetShortPathName to get guaranteed ASCII
wchar_t s_arr[4096];
if (0 == GetShortPathNameW(w_arr, s_arr, 4096)) {
delete [] w_arr;
return 0;
}
delete [] w_arr;
// wchar_t[] to ASCII nsEmbedString
nsEmbedString ascii;
wchar_t * s_arr_p = s_arr;
while (*s_arr_p) {
ascii.Append((char)*s_arr_p++);
}
// Macs don't need any help since they understand UTF-8
#else
nsEmbedString ascii;
ascii.Assign(fake);
#endif
// Convert the nsEmbedString into a std::string
char * c_arr = c_arr = new char[ascii.Length() + 1];
if (0 == c_arr) return 0;
char * c_arr_p = c_arr;
PRUnichar * c_start = (PRUnichar *)ascii.BeginReading();
const PRUnichar * c_end = (PRUnichar *)ascii.EndReading();
while (c_start != c_end) {
*c_arr_p++ = (char)*c_start++;
}
*c_arr_p = 0;
string * str = new string(c_arr);
delete [] c_arr;
return str;
}
This works simply because, on either platform, the result is a
char * that actually uniquely represents the desired file.
My code sucks, let me count the ways
There are of course things that could be better. First of all, after reading the code again (I wrote it Thursday and haven't looked back), I see that I should refactor a bit. By leaving the string in UTF-16 representation for passing to C-land, I can add a transform from UTF-16 to UTF-8 to the Mac version and greatly simplify the Windows version. I'll put that on my to-do list. The other place I hope to improve is in the event that
GetShortPathName fails. In this case, copying the file under a new ASCII filename to a temporary directory (found using
GetTempPath) would still let the file be accessed with a
char * path.
Why did I have to do this?
It would be sweeeeeet if Mozilla integrated this code (after a healthy dose of optimization) into the toolkit as something like
NS_ConvertToNativePath(nsAString) or some such function. Having this will make integrating with third-party libraries much easier.
Comments (7)
All inspirational credit, including the name, go to Aaron Straup Cope (filtr).
Curvr is a command-line tool that automates the most common operations I do in photoshop when I take pictures. I'm quite lazy and I hope that having this tool will make me take more pictures. This is the same line of reasoning that led me to an N73 and that is working out quite well.
I could just run Aaron's filtr script, which does a sort of Lomo-y thing, but that's not so much my style*. Instead, curvr perserves a little more reality and has a couple of key benefits. (* My style being slightly high-contrast but otherwise realistic.)
- Sets white- and black-points just like you can do in Photoshop's Levels dialog.
- Tweaks the color curve to make mid-tones just a touch darker and force more contrast.
The tweak of the color curve has a couple of nice side effects. Since it doesn't touch colors near the white- or black-points, things look pretty realistic. And for some reason, tweaking the mid-tones the way it does has the side effect of reducing haziness pretty effectively. Plus, skies pick up a really nice deep blue color.
My photography professor in college told us to use the red channel of the image to build black & white from color, rather than letting Photoshop convert it to greyscale. This advice carries through to the color mapping functions in curvr -- when replacing colors, the red channel is used as the key into the color map. It works surprisingly well.
Now the how-to-get-it bit. On a Mac you'll need MacPorts and a Terminal. On Linux you'll probably be able to pick a lot of this up from your package manager (update the Makefile accordingly).
$ sudo port install libpng jpeg
Now you can unpack and install GraphicsMagick. The GraphicsMagick source can be obtained at
ftp://ftp.graphicsmagick.org/pub/GraphicsMagick/. I use version 1.1.10.
./configure && make && sudo make install
Finally, curvr is just a simple
make.
curvr 0.1 http://svn.rcrowley.org/svn/curvr/ http://github.com/rcrowley/curvr
Comments (5)
Despite my blog being stale like the bread in my cabinet, I've still been elbow deep in XULRunner all day every day. Now 3 1/2 months in, the ugly stuff has taken shape and it's time to lock down the user interface. I still haven't found an elegant way to keep clicks on the scrollbar from firing my sequence of mousedown, mousemove and mouseup events, but gimme a second chance.
As expected, I'm learning a lot about releasing software in the real world, where it has to stand up to real revisions, new features and public scrutiny. I like to think I've always built software that's up to the task but a UI worthy of the Flickr name is a lot of work. And... Surprise! My engineer's touch is not good enough. Gino and George, with a healthy bit of Stewart, have made it look awesome. And my role in the design-y stuff? Implementing some of the ideas so we can play with them before springing them on unsuspecting users.
But stay tuned. It turns out that work not only fluctuates between C++ and JavaScript but between engineering (then) and design (now). After round one is out in the wild it will be time for more hacking in the great undocumented. Lather, rinse, repeat.
The XPCOM tips will be back soon!
Comments (0)
The majority of my (awesomest ever) day job at Flickr is written in Javascript. I do write some C++ code for some of the trickier parts, but wc tells me more that only about a tenth of my code is C++.
That tenth doesn't really do it for me, though. It isn't dirty and dangerous. Because of the Netscape Portable Runtime, it's actually rather tame. See, I spent college SEGFAULTing at least once a day. In my nearly three months at Flickr, I've managed to do so only twice.
Fear not -- I have a way to get my fix. I'm intrigued by FUSE and started to think about the design of a Flickr filesystem. About 14 seconds after that I discovered Manish Rai Jain's flickrfs project. It is more complicated than I would have designed, but because of that it's incredibly powerful. Flickr isn't out as a possible project for a FUSE filesystem but I'd rather not step on his toes. Onward.
The problem is, I'm short on imagination right now. I think the most powerful aspect of making non-filesystem data available as a filesystem is the huge set of possibilities that open up when you start chaining standard UNIX commands.
Do any C-shy webdev types want anything else out of their filesystem?
Comments (2)