Return application octet-stream data

[Noah Silverman]

Hello,

I am attempting to write a small CGI client in C++ that will use Google's protocol buffers to manage data. I am using the latest release of fast cig, with Nginx.

Google's protocol buffers library serializes a string into a "binary format". I need to be able to send that format back as a response. (Details on the format are here: https://developers.google.com/protocol-buffers/docs/encoding)

My test program sends an POST request to the server, and expects an application/octet-stream back of a serialized string from the CGI script.

I am able to start and run the application fine.

My CGI code is able to successfully receive the POST and pass the data fro the received string. That works great.

The problem is in returning back a response. I am able to create and serialized a proper response (using protobuf library). I can then dump the contents to a log file and verify that they are perfect. HOWEVER, the receiving program is unable to recognize the binary response sent back from the CGI. Somehow, it is getting mangled in sending the response. I can't figure out why.

Any and all suggestion are welcome.

Thank You,

--
Noah

Relevant code snippets:


CGI snippet
==========================================================
string post_response;
bid_response.SerializeToString(&post_response)
cout << "Status: 200 OK\r\n";
cout << "Content-type: application/octet-stream\r\n\r\n";
cout << post_response << "\n";


Nginx config
==========================================================
location ~ test.cgi$ {
fastcgi_pass 127.0.0.1:8000;
include fastcgi_params;
default_type application/octet-stream;
fastcgi_pass_header http;
}

[Noah Silverman]

Very possibly,

But since I don't control the serialization I have no way to manage that.

Is there a way to force fastcgi to ignore a zero byte?

--
Noah Silverman
Smart Media Corp.

Perhaps your serialized data contains a byte containing zero which is being taken as an end marker (prematurely terminating the response)?
Hello,
I am attempting to write a small CGI client in C++ that will use Google's protocol buffers to manage data. I am using the latest release of fast cig, with Nginx.
Google's protocol buffers library serializes a string into a "binary format". I need to be able to send that format back as a response. (Details on the format are here: https://developers.google.com/protocol-buffers/docs/encoding)
My test program sends an POST request to the server, and expects an application/octet-stream back of a serialized string from the CGI script.
I am able to start and run the application fine.
My CGI code is able to successfully receive the POST and pass the data fro the received string. That works great.
The problem is in returning back a response. I am able to create and serialized a proper response (using protobuf library). I can then dump the contents to a log file and verify that they are perfect. HOWEVER, the receiving program is unable to recognize the binary response sent back from the CGI. Somehow, it is getting mangled in sending the response. I can't figure out why.
Any and all suggestion are welcome.
Thank You,
--
Noah
CGI snippet
==========================================================
string post_response;
bid_response.SerializeToString(&post_response)
cout << "Status: 200 OK\r\n";
cout << "Content-type: application/octet-stream\r\n\r\n";
cout << post_response << "\n";
Nginx config
==========================================================
location ~ test.cgi$ {
fastcgi_pass 127.0.0.1:8000;
include fastcgi_params;
default_type application/octet-stream;
fastcgi_pass_header http;
}
_______________________________________________
FastCGI-developers mailing list
http://mailman.fastcgi.com/mailman/listinfo/fastcgi-developers
--
---
"Why do you have that banana in your ear?"
"To keep away the alligators."
"But there are no alligators here."
"See! It's working!"
--
The Democrats are the left wing.
The Republicans are the right wing.
We all know how well a bird flies with just one wing.

[Charles Thomas]

From what I understand of HTTP protocol Content-Length is used to determine how long the message is... not the occurrence of a null character in the data stream; and that content length starts after the newline separating headers from data.

Some thoughts:

1) Did you separate the headers with a "\n" from the body
2) Is the content length truly the length of the data
3) Does the mime type have to be set correctly in order for server you are talking with to recognize,
4) Some times I have found that servers expect the Accept tag to contain specific reference to the type of file.


Sent from my iPad

Very possibly,
But since I don't control the serialization I have no way to manage that.
Is there a way to force fastcgi to ignore a zero byte?
--
Noah Silverman
Smart Media Corp.

Perhaps your serialized data contains a byte containing zero which is being taken as an end marker (prematurely terminating the response)?
Hello,
I am attempting to write a small CGI client in C++ that will use Google's protocol buffers to manage data. I am using the latest release of fast cig, with Nginx.
Google's protocol buffers library serializes a string into a "binary format". I need to be able to send that format back as a response. (Details on the format are here: https://developers.google.com/protocol-buffers/docs/encoding)
My test program sends an POST request to the server, and expects an application/octet-stream back of a serialized string from the CGI script.
I am able to start and run the application fine.
My CGI code is able to successfully receive the POST and pass the data fro the received string. That works great.
The problem is in returning back a response. I am able to create and serialized a proper response (using protobuf library). I can then dump the contents to a log file and verify that they are perfect. HOWEVER, the receiving program is unable to recognize the binary response sent back from the CGI. Somehow, it is getting mangled in sending the response. I can't figure out why.
Any and all suggestion are welcome.
Thank You,
--
Noah
CGI snippet
==========================================================
string post_response;
bid_response.SerializeToString(&post_response)
cout << "Status: 200 OK\r\n";
cout << "Content-type: application/octet-stream\r\n\r\n";
cout << post_response << "\n";
Nginx config
==========================================================
location ~ test.cgi$ {
fastcgi_pass 127.0.0.1:8000;
include fastcgi_params;
default_type application/octet-stream;
fastcgi_pass_header http;
}
_______________________________________________
FastCGI-developers mailing list
http://mailman.fastcgi.com/mailman/listinfo/fastcgi-developers
--
---
"Why do you have that banana in your ear?"
"To keep away the alligators."
"But there are no alligators here."
"See! It's working!"
--
The Democrats are the left wing.
The Republicans are the right wing.
We all know how well a bird flies with just one wing.

_______________________________________________
FastCGI-developers mailing list
http://mailman.fastcgi.com/mailman/listinfo/fastcgi-developers

[Noah Silverman]

Charles,

If you'll notice, in my C++ snippet, I am outputting a double carriage return between the headers and the body:

bid_response.SerializeToString(&post_response)
cout << "Status: 200 OK\r\n";
cout << "Content-type: application/octet-stream\r\n\r\n";
cout << post_response << "\n";


--
Noah

From what I understand of HTTP protocol Content-Length is used to determine how long the message is... not the occurrence of a null character in the data stream; and that content length starts after the newline separating headers from data.
1) Did you separate the headers with a "\n" from the body
2) Is the content length truly the length of the data
3) Does the mime type have to be set correctly in order for server you are talking with to recognize,
4) Some times I have found that servers expect the Accept tag to contain specific reference to the type of file.
Sent from my iPad

Post by Noah Silverman
Very possibly,
But since I don't control the serialization I have no way to manage that.
Is there a way to force fastcgi to ignore a zero byte?
--
Noah Silverman
Smart Media Corp.

Perhaps your serialized data contains a byte containing zero which is being taken as an end marker (prematurely terminating the response)?
Hello,
I am attempting to write a small CGI client in C++ that will use Google's protocol buffers to manage data. I am using the latest release of fast cig, with Nginx.
Google's protocol buffers library serializes a string into a "binary format". I need to be able to send that format back as a response. (Details on the format are here: https://developers.google.com/protocol-buffers/docs/encoding)
My test program sends an POST request to the server, and expects an application/octet-stream back of a serialized string from the CGI script.
I am able to start and run the application fine.
My CGI code is able to successfully receive the POST and pass the data fro the received string. That works great.
The problem is in returning back a response. I am able to create and serialized a proper response (using protobuf library). I can then dump the contents to a log file and verify that they are perfect. HOWEVER, the receiving program is unable to recognize the binary response sent back from the CGI. Somehow, it is getting mangled in sending the response. I can't figure out why.
Any and all suggestion are welcome.
Thank You,
--
Noah
CGI snippet
==========================================================
string post_response;
bid_response.SerializeToString(&post_response)
cout << "Status: 200 OK\r\n";
cout << "Content-type: application/octet-stream\r\n\r\n";
cout << post_response << "\n";
Nginx config
==========================================================
location ~ test.cgi$ {
fastcgi_pass 127.0.0.1:8000;
include fastcgi_params;
default_type application/octet-stream;
fastcgi_pass_header http;
}
_______________________________________________
FastCGI-developers mailing list
http://mailman.fastcgi.com/mailman/listinfo/fastcgi-developers
--
---
"Why do you have that banana in your ear?"
"To keep away the alligators."
"But there are no alligators here."
"See! It's working!"
--
The Democrats are the left wing.
The Republicans are the right wing.
We all know how well a bird flies with just one wing.

_______________________________________________
FastCGI-developers mailing list
http://mailman.fastcgi.com/mailman/listinfo/fastcgi-developers

[Jay Sprenkle]

Glad that helped.
I recall trying to get fastcgi working and being very frustrated.

Jay,
That did it!!!
Thank you so much!
--
Noah
bid_response.SerializeToString(&post_response)
cout << "Status: 200 OK\r\n";
cout << "Content-type: application/octet-stream\r\n";
cout << "Content-Length: add-your-response-length here\r\n\r\n"
cout.write( post_response, add-your-response-length here );
cout << "\n";

[Charles Thomas]

i guess it would help if I read the entire mail message :)... so it was the missing Content-Length?


Sent from my iPad

Post by Jay Sprenkle
Glad that helped.
I recall trying to get fastcgi working and being very frustrated.
Jay,
That did it!!!
Thank you so much!
--
Noah

Post by Noah Silverman
bid_response.SerializeToString(&post_response)
cout << "Status: 200 OK\r\n";
cout << "Content-type: application/octet-stream\r\n";
cout << "Content-Length: add-your-response-length here\r\n\r\n"
cout.write( post_response, add-your-response-length here );
cout << "\n";

_______________________________________________
FastCGI-developers mailing list
http://mailman.fastcgi.com/mailman/listinfo/fastcgi-developers

[Jay Sprenkle]

My guess is that was the issue.
The other (terminating the string at the first zero byte) might have caused
similar problems as well.
Computers are so fussy ;)

Post by Charles Thomas
i guess it would help if I read the entire mail message :)... so it was
the missing Content-Length?

[Charles Thomas]

Yeah... it's a real love-hate relationship for me. I love fixing the problems that I Hate getting.

The weirdest one I have had with FastCGI was cross-domain transfer from a javascript to fastcgi. With that one there were some weird requirements concerning what headers I could and could not have in the request/response.

Speaking of headers and fastCGI, are there any classes that encapsulate the header logic into them? I know about the fastcgi library, and one for Qt (FastCgiQt), but I was wondering if there are any more higher level classes out there?

Tom

Sent from my iPad

Post by Jay Sprenkle
My guess is that was the issue.
The other (terminating the string at the first zero byte) might have caused similar problems as well.
Computers are so fussy ;)
i guess it would help if I read the entire mail message :)... so it was the missing Content-Length?

[Noah Silverman]

Hi,

It looks like I'm still having a bit of trouble.

Thanks to Jay's help, I can reliably return a binary string from the fastcgi program back to the webserver(and remote connection).

I've noticed that about 50% of the *incoming* requests aren't getting parsed properly. All incoming requests are sent as post data of an octet-stream. My guess is that the way I'm reading them in isn't happy with a null character or something similar.

Any ideas on how to fix this one? I imagine that there is a "best practices" way to safely read in the post data from nginx.

Thanks!

Here is the c++ snippet:
(Taken from the example script supplied with fastcgi.)

============================================
static long gstdin(FCGX_Request * request, char ** content){
char * clenstr = FCGX_GetParam("CONTENT_LENGTH", request->envp);
unsigned long clen = STDIN_MAX;
if (clenstr){
clen = strtol(clenstr, &clenstr, 10);
if (*clenstr){
cerr << "can't parse \"CONTENT_LENGTH="
<< FCGX_GetParam("CONTENT_LENGTH", request->envp)
<< "\"\n";
clen = STDIN_MAX;
}

// *always* put a cap on the amount of data that will be read
if (clen > STDIN_MAX) clen = STDIN_MAX;

*content = new char[clen];

cin.read(*content, clen);
clen = cin.gcount();
}
else{
// *never* read stdin when CONTENT_LENGTH is missing or unparsable
*content = 0;
clen = 0;
}

// Chew up any remaining stdin - this shouldn't be necessary
// but is because mod_fastcgi doesn't handle it correctly.

// ignore() doesn't set the eof bit in some versions of glibc++
// so use gcount() instead of eof()...
do cin.ignore(1024); while (cin.gcount() == 1024);
return clen;
}

int main (void){

// Backup the stdio streambufs
streambuf * cin_streambuf = cin.rdbuf();
streambuf * cout_streambuf = cout.rdbuf();
streambuf * cerr_streambuf = cerr.rdbuf();

FCGX_Request request;

FCGX_Init();
FCGX_InitRequest(&request, 0, 0);

//The main loop for fast cgi
while (FCGX_Accept_r(&request) == 0){
char * content;
unsigned long clen = gstdin(&request, &content);
etc…
….
}
return 0;
}
============================================

Post by Charles Thomas
Yeah... it's a real love-hate relationship for me. I love fixing the problems that I Hate getting.
The weirdest one I have had with FastCGI was cross-domain transfer from a javascript to fastcgi. With that one there were some weird requirements concerning what headers I could and could not have in the request/response.
Speaking of headers and fastCGI, are there any classes that encapsulate the header logic into them? I know about the fastcgi library, and one for Qt (FastCgiQt), but I was wondering if there are any more higher level classes out there?
Tom
Sent from my iPad

Post by Jay Sprenkle
My guess is that was the issue.
The other (terminating the string at the first zero byte) might have caused similar problems as well.
Computers are so fussy ;)
i guess it would help if I read the entire mail message :)... so it was the missing Content-Length?

[Jay Sprenkle]

I don't know of any but I really haven't looked for them.
I believe the various web servers have differences in how they parse the
responses.
It took forever for me to dig out some of the information about what Apache
expects to receive back from a cgi program.

cgicc does html and forms handling:
http://www.gnu.org/s/*cgicc*

Post by Charles Thomas
Yeah... it's a real love-hate relationship for me. I love fixing the
problems that I Hate getting.
The weirdest one I have had with FastCGI was cross-domain transfer from a
javascript to fastcgi. With that one there were some weird requirements
concerning what headers I could and could not have in the request/response.
Speaking of headers and fastCGI, are there any classes that encapsulate
the header logic into them? I know about the fastcgi library, and one for
Qt (FastCgiQt), but I was wondering if there are any more higher level
classes out there?
Tom
Sent from my iPad
My guess is that was the issue.
The other (terminating the string at the first zero byte) might have
caused similar problems as well.
Computers are so fussy ;)

Post by Charles Thomas
i guess it would help if I read the entire mail message :)... so it was
the missing Content-Length?

--
---
*"Why do you have that banana in your ear?"
"To keep away the alligators."
"But there are no alligators here."
"See! It's working!"*
*--*
*The Democrats are the left wing.*
*The Republicans are the right wing.*
*We all know how well a bird flies with just one wing.*

[Noah Silverman]

Jay,

Other way around.

Thanks to your help, my fastcgi program now *returns* the binary data perfectly.

My new challenge is that it doesn't always properly read the input of binary data.

What would be a good way to setup the correct file handle to read in the binary post data?

--
Noah

Post by Jay Sprenkle
I don't know of any but I really haven't looked for them.
I believe the various web servers have differences in how they parse the responses.
It took forever for me to dig out some of the information about what Apache expects to receive back from a cgi program.
http://www.gnu.org/s/cgicc
Yeah... it's a real love-hate relationship for me. I love fixing the problems that I Hate getting.
The weirdest one I have had with FastCGI was cross-domain transfer from a javascript to fastcgi. With that one there were some weird requirements concerning what headers I could and could not have in the request/response.
Speaking of headers and fastCGI, are there any classes that encapsulate the header logic into them? I know about the fastcgi library, and one for Qt (FastCgiQt), but I was wondering if there are any more higher level classes out there?
Tom
Sent from my iPad

Post by Jay Sprenkle
My guess is that was the issue.
The other (terminating the string at the first zero byte) might have caused similar problems as well.
Computers are so fussy ;)
i guess it would help if I read the entire mail message :)... so it was the missing Content-Length?

--
---
"Why do you have that banana in your ear?"
"To keep away the alligators."
"But there are no alligators here."
"See! It's working!"
--
The Democrats are the left wing.
The Republicans are the right wing.
We all know how well a bird flies with just one wing.