Attendees: ashrader330, clsk, j_k9, tt

Note: All timestamps in UTC.

This meeting did not have an agenda.

As sent to the Mira Development Mailing List on 16/12/2010 with the subject: “Summary of 16/12/2010 Meeting”:

Hi all,

Tonight's meeting was attended by Adam, Alan, Tijoy and myself. You can find a transcript of the meeting here:
http://miragroupware.org/wiki/doku.php/development/communication/archives/101216_message_center

I shall address the topics in the order in which they were discussed. Please use this mailing list thread to discuss these topics: many of the issues raised in the meeting were still not decided at the end of it, so this is the place to debate them before our next meeting.

For the Message Center, do we implement a custom messaging system or use a “standard” backend?
Adam said that this depends on what we want to do. For example, if we want to be able to send emails to other systems, we should use the “standard” SMTP/Exchange with IMAP/POP3/Exchange. Alternatively, if we do not want this functionality, it is easier to keep the messaging system contained and build it ourselves.
I said that our aim is not to replace the user's email client; so we should start with a contained system. We want to support rich text and attachments, but we do not want to interact with alternative systems.
The best option, which was agreed on, is to implement our own custom messaging system with a “framework” approach so that, in the future, if we wish to support the other “standard” backends, we can extend the framework to support them.

What to use for network message structure and content (including serialization)?
A few options were suggested:
Adam said that using XML might make it easier to validate messages. It is easier to understand and debug.
Tijoy proposed JSON as an option because of its benefits in low-bandwidth scenarios. Alan said that we might need something more verbose than JSON for a mailing system of this kind.
If bandwidth is a concern, Adam suggested we look in to using protocol buffers, such as Google Protocol Buffers, for a very efficient data serialization format. Tijoy added that, if we use protocol buffers, we don't have to worry about object→XML/JSON and XML/JSON→object conversion because the protocol buffer generates the stubs itself. However, he also expressed a concern that protocol buffers would give us more than we need, e.g. different langage bindings (we don't use Java or Python).
Adam has used a message passing broker system in the past: QPID (an AMQP broker), with Qt. He found that it served all his requirements.

It was suggested that if Adam started to work on the Message Center, Tijoy could then join in once he is more familiar with Mira's code. The tasks for this feature could then be divided accordingly.

Alan raised the point that the Network layer is not binary-friendly, in that every “\r\n” or “\n” is treated as the beginning of a new message.
In order to support the Message Center, the Network layer will have to be modified to support binary data. As Alan wrote the existing Network layer code, he suggested that the three of them could work on restructuring this Layer together, at least at the beginning, to upgrade it in line with the requirements of the Message Center.
The existing network messages that we use could be “upgraded” for compatibility with the new network system with little difficulty. Adam said that we could encode their information with any of the proposed data serialization methods. Alan said that our existing messages can easily be adapted to the way messages are defined in protocol buffers.
Adam proposed that one field within the network message could be designated as the message type. That message type would then allow the code to interpret the rest of the fields appropriately.

Adam is going to prepare some code examples and submit them to the group for discussion. I asked him to publish them to a personal branch on Launchpad. He says he might be able to do some of it this weekend.

At the end of the meeting, Tijoy said that we should consider using HTTP, because TCP on a non-standard port (as we currently use) is often blocked in an enterprise environment. A problem put forward by Adam is that it might conflict with normal web traffic.

Discuss. :)

Regards,

Max Bossino
Project Manager
http://miragroupware.org

23:47

J_K9: Hey clsk - logged in on my phone, in case anything happens while on my way back :)

ashrader330: Hey Max. I'm here as well.

J_K9: Hey Adam, glad you could join us!

23:48

ashrader330: Not a problem.

clsk: hello

J_K9: If Alan is here, feel free to be begin - I should be getting home in about fifteen minutes

J_K9: Hey Alan :)

clsk: I'm sorry I didn't answer last night Adam, I wasn't at home

ashrader330: Ok. I have been looking into what Alan mentioned about using a home-grown messaging system versus a standard one like IMAP and it really matters about what you want to do with this messaging system.

23:51

clsk: Can you explain further Adam?

ashrader330: If you want this to send e-mails to other systems, then we use IMAP. Otherwise creating something from scratch would be easiest. IMAP is only the retreval part of a e-mail system. If we were to use that then we would have to use SMTP to send messages.

clsk: right

J_K9: I think the idea is to have a contained system. We don't want to pretend to replace the email client. However, we do want our system to support rich text, attachments etc

23:55

ashrader330: I only found 1 IMAP library that is still actively developed also. Max, all of that can easily be done with XML so that would not be an issue.

clsk: I think the best way (feature-wise) to implement it, is to support different “connectors” for different types of protocols/systema. Such as IMAP, POP3 and exchange

clsk: that's for mail retrieving messages

clsk: for sending messages probably only SMTP and maybe exchange protocol would be enough

clsk: the reason why I think this is appealing to users is that they wouldn't have to migrate to a totally new mailing system

J_K9: My reasoning, clsk, is that this would operate alongside their existing mailing system

J_K9: I.e the internal mailing system would be strictly related to matters pertinent to their Mira environment

23:58

J_K9: I see the benefit of your framework approach - it is very flexible. But is it necessary for what we want to accomplish?

clsk: I don't know

clsk: really I'm just proposing an idea

clsk: I think it will be just as difficult to create our own mailing system from scratch

J_K9: What do you think, Adam? What's your view?

ashrader330: Alan, I kind of had that idea too but I wasn't sure what the intention of the messaging system. I feel like it should be all of the above. They all have a use.

00:02

ashrader330: It wouldn't be too hard to create something some scratch. It would be more difficult to make a IMAP-standard compliant client in my opinion. I looked over the standard and it is 109 pages long.

clsk: then we should probably start from scratch

ashrader330: Again, I think eventually giving the users an option about what backend to use would be best.

clsk: http:/​/​www.vmime.org/

clsk: a library like that should make it easy

ttk joined the room.

00:05

J_K9: Sorry, hopped in a lift - did I miss anything after Alan's link to vmime?

clsk: not really

ashrader330: Yeah, I found that library yesterday. That is the one that I mentioned when I said I could only find one library.

tijoytk_ joined the room.

clsk: who's ttk?

J_K9: What if we start from scratch with a framework-type approach, then make it more friendly to other libraries in a later release?

ashrader330: Don't know.

J_K9: Hey Tijoy :)

clsk: well if you think it's better to work on our own system from scratch then we should at least give it a try

00:09

J_K9: Wouldn't a project like Pigeon make use of some cross-platform, Open Source mail libraries?

clsk: I think they have their own

ashrader330: Are you talking about Pidgin, the chat client?

J_K9: Sorry, it's been too long :( I think I'm talking about Thunderbird

tijoytk_ left the room. (Ping timeout: 240 seconds)

J_K9: (I knew there was a bird in there…)

00:12

ashrader330: :) Close enough. Thunderbird definately cooked up their own code on that one.

j_k9_ joined the room.

[00:16] j_k9_: Just switched to laptop - testing.

[00:16] J_K9 left the chat room. (Quit: J_K9)

[00:16] You are now known as j_k9.

[00:16] j_k9: Ok, better :)

[00:17] clsk: alright, so are we deciding for our own mailing system for now?

[00:18] ttk joined the chat room.

[00:18] j_k9: I suppose so! Adam, have you thought about how you might implement such a system? XML, by the sound of it :)

[00:19] ashrader330: For now, but I will keep the design flexible enought to include other backends. Yes, because that is how much of it is done and it would allow us to validate messages when they are received.

[00:20] ashrader330: Unless somebody else has an objection.

[00:21] j_k9: No objection on my part; Alan used XML for the text-based Directory backend, and although this is a different kettle of fish I'd be happy to say yes.

[00:21] clsk: no objections

[00:21] clsk: there's one thing you should note though

[00:21] clsk: the network protocol is not binary-friendly

[00:22] ashrader330: How so?

[00:22] ttk left the chat room. (Read error: Connection reset by peer)

[00:23] ttk joined the chat room.

[00:23] clsk: the reason why I say this, is that everytime there is a “\r\n” or a '\n', the server and client will treat this as a new message

[00:24] clsk: one option is to modify the network protocol (this might take some work, but might be worth it in the end), or just work around it

[00:25] ashrader330: Where would that code be at?

[00:25] clsk: A member of TcpConnection on the server

[00:25] tijoytk_ joined the chat room.

[00:25] clsk: can't remember on the client

[00:26] clsk: but shouldn't be hard to find

[00:26] clsk: I can give you the exact location if needed

[00:26] tijoytk_ left the chat room. (Client Quit)

[00:27] clsk: probably the best to do it would be to use pre-configured network packets

[00:27] ashrader330: I will look around. I will try to do that.

[00:28] ttk left the chat room. (Ping timeout: 265 seconds)

[00:29] j_k9: Sorry Adam, one more thing: do you have any criticisms or suggestions on the Message Center blueprint? i.e. feature-wise?

[00:29] tt joined the chat room.

[00:30] ashrader330: No, I think it is pretty good.

[00:31] j_k9: I suppose that is a good start!

[00:31] clsk: ashrader330: I've worked around passing xml-formatted message using the network protocol

[00:31] clsk: messages*

[00:31] clsk: if you look at the files utility that's what I use for the filesystem layout

[00:32] clsk: but I use a different port transferring files

[00:33] ashrader330: Is that under the trunk or your baseline?

[00:33] clsk: however, getting rid of that “issue” is something that will need to happen in the future regardless

[00:33] clsk: hm I think it's on trunk as well

[00:33] clsk: but definitely on my branch

[00:34] ashrader330: Is there any way to see other users' branches?

[00:34] j_k9: You can get it with Bazaar, e.g.: bzr branch lp:~mira-dev/mira/login-system

[00:34] j_k9: (I think that's the reference to Alan's)

[00:35] clsk: you can look at it on launchpad

[00:35] clsk: https://code.launchpad.net/mira

[00:35] clsk: those are all the registered branches

[00:35] clsk: login-system is the on I'm working on atm.

[00:38] tt: Hi All , This's tijoy . .. Sorry to chime in now … I was reading the discussion about network protocol and xml …

[00:38] tt: Kind of wondering JSON was considered ….

[00:39] tt: But JSON encoded strings might be a better choice for high latency server..

[00:40] j_k9: I think this did come up a while ago, because I have two links lying around: http://www.json.org/xml.html http://www.codeproject.com/KB/recipes/JSON_Spirit.aspx

[00:40] clsk: tt: I was thinking about just a few minutes ago

[00:40] clsk: it might be a better option

[00:40] j_k9: But I cannot remember what we decided… clsk didn't you consider JSON for the Directory layer?

[00:41] clsk: j_k9: Definitely not for the directory layer

[00:42] tt: I dont have anything against XML , it can become quite heavy especially in low bandwidth scenarios

[00:42] clsk: however, for something like a mailing system, we might need something more verbose than JSON

[00:43] tt: That's true…

[00:43] clsk: We should study the pros and cons of using both

[00:47] ashrader330: If bandwidth is a concern, there is always google protocol buffers. They are the most efficient data serialization format I have seen. It requires another library dependency.

[00:50] j_k9: Just taken a look at Google Protocol Buffers, seems very powerful. I don't want to spam this channel, but a short excerpt won't cause much harm:

[00:50] j_k9: Protocol buffers have many advantages over XML for serializing structured data. Protocol buffers:

[00:50] j_k9: - are simpler

[00:50] j_k9: - are 3 to 10 times smaller

[00:50] j_k9: - are 20 to 100 times faster

[00:50] j_k9: - are less ambiguous

[00:50] j_k9: - generate data access classes that are easier to use programmatically

[00:51] clsk: hm do we really need something that powerful though?

[00:51] clsk: good thing about JSON and xml is that they're simple

[00:51] clsk: and wouldn't really require a another library dependency

[00:54] tt: I thyink we need to distinguish between protocol buffer and protocol…

[00:54] tt: think*

[00:54] ashrader330: They are simple to use. I used them for a while in my day job. They do have some quirks to work around but they are, for the most part, pretty easy to use.

[00:55] tt: Protocol buffers help serializing /deserializing data….while the underlying high protocol can be JSON or XML

[00:55] tt: example , you can use google protocol buffers with JSON , XML or binary

[00:58] tt: Another protocol buffer implemention that came up is Thrift….

[00:58] ashrader330: Serializing data is what JSON and XML do. JSON, XML, and GPB all do the same thing different ways.

[01:03] clsk: I think what we need is a good balance between simple and effective

[01:04] tt: If we use protocol buffers , we dont have to worry about object→ xml (JSON) and xml(JSON)→object conversion … Protocol buffer generate the stubs for you …

[01:04] ashrader330: Protocol buffer will do that.

[01:06] clsk: so if we choose to go that way, probably the best thing is to modify the whole network protocol and current code to use this

[01:06] tt: But on the other hand if we decide to use JSON or XML without protocol buffer we will have to code XML - > object and object - XML

[01:06] clsk: who's up for that task?

[01:07] tt: I guess that we will some kind of collabrative editing feature in future …

[01:07] tt: I can help ….

[01:07] tt: But it might take some time for me to ramp up

[01:07] ashrader330: I wouldn't mind because it will affect what I do on the messaging utility. Plust I do have experience with them.

[01:09] tt: If Adam can start , i can start contibuting once i get some idea about current code

[01:09] clsk: well the first thing we need to do is design a good architecture and figure out how much restructuring the current code will need

[01:10] tt: Agreed.

[01:10] ashrader330: Agreed. Coding is easy. Design is hard.

[01:12] clsk: well since I know my way around the current code, and you guys have good ideas, would probably best the 3 of us work on this together at least to get started, and then we can assign different tasks

[01:13] tt: I am fine with it …

[01:13] ashrader330: Sounds good to me.

[01:13] clsk: do you guys want to start on the overall general design today?

[01:14] ashrader330: Sure.

[01:18] j_k9: Well… where to begin? Alan, you know the code best, seeing that you wrote it! :) Shall we start with what you think needs to be replaced?

[01:18] clsk: I'm currently reading descriptions for protocol buffers and Thrift

[01:28] clsk: hm what do you guys think is a better option?

[01:31] ashrader330: Thrift and protocol buffers do much of the same thing the same way. One was developed by Facebook and the other by Google. Protocol buffers are more established and more mature than Thrift.

[01:32] tt: Is google protocol buffer open source ?

[01:32] ashrader330: XML is more verbose and can do more message validation. It is easiesr to understand and debug.

[01:32] ashrader330: Yes, it is released under LGPL.

[01:37] clsk: I see a lot of (probably not very effective) getting removed in the near future

[01:37] tt: I think using either google protocol buffers or Thrift woulbe be better … but the learning curve is high compared to using plain XML or JSON

[01:40] tt: Even if we decide to use XML or JSON , as project grow ( more features) we will end writing some kind of protocol buffer…

[01:41] clsk: I'm reading through protocol buffers and it seems pretty straight-forward for the most part

[01:43] ashrader330: They are very easy to use. The only issue I have had with them was sending binary data from a QByteArray through protocol buffers didn't always translate as I expected. Just took a little more work to it working correctly.

[01:47] tt: My only concern with protocol buffer is it gives us more than what we really need

[01:49] ashrader330: How so?

[01:50] tt: It has lot of features which we might not need…

[01:50] ashrader330: Now maybe but what about in the future?

[01:51] tt: Ya , we might use them ..

[01:52] clsk: can you say what some of those features are?

[01:52] clsk: I've only known about this library for a few minutes

[01:52] clsk: just curious

[01:55] tt: say different language bindings…

[01:55] tt: We dont need python or java i guess

[01:56] ashrader330: The features I have used it provides binary data serialization, encodes data into a standard C++ string, provides a means of validating messages to ensure all fields are there.

[01:56] ashrader330: Automated code generation.

[02:02] clsk: well in the current code there is a Msg parent class of which all the other Message parsers derive from

[02:03] clsk: what's the best way of reading the network packets and then figuring out what type of message you get?

[02:08] tt left the chat room. (Quit: Page closed)

[02:08] ashrader330: I have used a message passing broker system i.e. AMQP broker (QPID). With Qt. I haven't found anything I have not been able to do with Qt and QPID.

[02:18] clsk: what exactly is that for?

[02:21] j_k9: “http://www.amqp.org/ is the first open standard for Enterprise Messaging. It is designed to support messaging for just about any distributed or business application.” I'm surprised we haven't come across this before!

[02:21] tt joined the chat room.

[02:21] j_k9: QPID here: http://qpid.apache.org/

[02:22] ashrader330: It is a message passing system. It routes messages between different “mailboxes” given a “routing key”

[02:22] tt: Yes , it's like Microsoft MSMQ

[02:23] j_k9: Adam, you say you've worked with this before? Would it be able to support what is required by the Message Center (and maybe even the Files Utility)?

[02:23] j_k9: (In other words, all types of data.)

[02:23] clsk: hm I don't see why we'd need this for the overall network protocol. Or are you now talking about the mailing system in particular?

[02:24] ashrader330: Yeah, we use it for practically everything. Interprocess communication, communication between processes, between workstations, etc.

[02:25] ashrader330: I meant interthread communication not interprocess.

[02:26] clsk: hm you guys are confusing me

[02:26] ashrader330: Sorry, what can I clear up?

[02:27] clsk: well, we were talking about the network protocol for the client↔server communication

[02:27] clsk: and now we're talking about the mailing system

[02:27] ashrader330: I thought we were still talking about the network system.

[02:27] clsk: or are we still talking about the overall network protocol?

[02:28] clsk: oh ok.. got what you mean now

[02:28] clsk: I don't think we need something like that

[02:28] ashrader330: When I was describing the AMQP standard, I was using their terms for how things work.

[02:28] clsk: ah ok

[02:30] clsk: yea, I guess something like that is what we need

[02:30] clsk: Internally that is though

[02:31] clsk: not over the network

[02:32] ashrader330: I am just a fan of the QPID implementation of the AMQP standard. I like the way it gives a standard way of passing information around.

[02:36] clsk: anyway… what we need to do is create messages for all the network messages we support so far

[02:36] clsk: define would be the word I guess

[02:44] ashrader330: How so?

[02:46] clsk: hm.. what?

[02:47] ashrader330: I don't exactly understand what you mean by create messages for all fo the network messages.

[02:48] j_k9: Would it not be possible to use the existing messages, but obviously adapt their implementations to the new system? http://miragroupware.org/wiki/doku.php/development/blueprints/net_protocols

[02:51] clsk: I'm sorry I didn't really explain myself

[02:51] ashrader330: I think it would. We could encode that information with any of the data serialization methods we mentioned.

[02:51] clsk: right now the way our network code works is, we have pre-defined “messages” that we pass between client and server

[02:51] clsk: right

[02:53] clsk: good thing is what we're using right now can be easily adapted to the way messages are defined in protocol buffers

[02:53] ashrader330: I think so too.

[02:55] clsk: my question is, when you have different types of messages, how can you tell from the data you receive from you remote point which type of message it is you are receiving? Does the library provide some type of API for this?

[02:55] clsk: method/mechanism

[02:56] clsk: I'm sure there's some type of metadata that travels with the data to figure this out

[02:56] clsk: just wondering if it's something that's made easy by the library

[03:04] ashrader330: Yes, one field could be the message type. That message type could then allow the code to interpret the rest of the fields.

[03:07] j_k9: That would resolve the \r\n issue and pave the way for the Message Center, right?

[03:07] clsk: yes

[03:11] clsk: well I guess that's one idea.. if you guys would like to take your time and study the code

[03:11] clsk: and then have another meeting to discuss the design and changes that need to be made let me know

[03:12] tt: sounds good….

[03:12] ashrader330: I can come up with some code examples and submit it to the group for discussion.

[03:13] tt: That would be great….

[03:13] j_k9: Ok, can I just make an attempt to summarise what we have discussed, because a lot of acronyms unfamiliar to me have been thrown around:

[03:13] j_k9: In support of the Message Center:

[03:13] j_k9: - Rewrite the network layer to support binary data

[03:13] j_k9: - This might involve using QPID with Qt

[03:13] j_k9: - The existing messages will need to be adapted to the new system (with one field designated as the “message type”)

[03:13] j_k9: With regard to the Message Center:

[03:13] j_k9: - We are going to create our own system

[03:13] j_k9: - But we are going to make it flexible enough to be capable of supporting new backends in the future

[03:13] j_k9: - We are going to consider using JSON, if not XML.

[03:13] j_k9: Have I said anything untrue, or left anything crucial out that we have discussed?

[03:14] j_k9: e.g. Protocol buffers?

[03:14] clsk: - This might involve using QPID with Qt → not really

[03:14] ashrader330: No, I don't think so.

[03:14] clsk: Adam just made a reference to AMQP and qPID

[03:15] j_k9: Ah, right, in terms of passing information

[03:15] clsk: - We are going to consider using JSON, if not XML….. I think our best option is to use protocol buffers which I think uses its own encoding

[03:16] clsk: but I guess it's still up for discussion

[03:17] j_k9: Ok, right. Sorry, so with regard to AMQP, is it just the structure of the messages that we would like to replicate?

[03:17] j_k9: In other words, just modify the Network Layer to follow that structure, and support binary data in the process?

[03:17] clsk: I guess just the general idea behind it

[03:18] j_k9: Hmm, ok. Adam, if you could come up with those code examples and push them to a new branch, that would be fantastic! :)

[03:19] j_k9: At least, it would help me see the woods from the trees :P And if it helps the three of you decide between the different proposals, so much the better.

[03:21] ashrader330: Will do. I might be able to get some of it this weekend.

[03:21] tt: before we close , should be consider using HTTP in future

[03:21] tt: Now as i know we use TCP , and use a non standard port

[03:22] tt: But in an enterprise envirnoment , usually non standard ports are blocked….

[03:24] j_k9: Could the Server not be run on port 80, for example? 7351 is the default port, but not the only one possible (I believe it can be configured in mira.conf, although I'm not sure if it's currently hard-coded).

[03:29] ashrader330: Not sure how it would work. It would have to not interfere with normal web traffic and not be confused by it either.

[03:35] j_k9: Well, shall we do something? Shall we (and by “we” I include myself, despite my limited technical knowledge) have a think about all the issues raised, and then hold another meeting when we're more aware of all the materials?

[03:36] j_k9: That way we can have a richer discussion. At the moment, we're getting lots of very useful ideas, but it looks like we're having a hard time choosing between them without knowing any more about them. :)

[03:37] j_k9: I'll send (my best attempt at) a summary of what we have discussed so far on the mailing list, and I will post a transcript on the wiki. Shall we share relevant information and thoughts on the mailing list, then convene a meeting to make the decisions?

[03:37] tt: Sure , that's the best way to move forward

[03:38] ashrader330: Yes, I would agree.

[03:38] j_k9: Great. :)

[03:39] clsk: yes

[03:39] j_k9: Ok, well thank you all for joining! I shall post up a transcript and send round the summary shortly. Feel free to reply to that thread on the mailing list to flesh out ideas, and your own opinions.

[03:40] j_k9: And, for the record, our meetings are not usually this long, so I hope you are not put off for next time! :P

[03:42] tt: sounds good.. Good night all…

[03:42] j_k9: Goodnight!

[03:42] ashrader330: Night.

[03:43] tt left the chat room. (Quit: Page closed)

[03:43] clsk: good night