Attendees: badri, clsk, j_k9, shilpa

Note: All timestamps in UTC+2

This meeting's agenda is available here.

To be added.

[00:57] shilpa joined the chat room.

[00:57] j_k9: Hi Shilpa :)

[00:58] shilpa: Hello.

[00:58] j_k9: Alan is about to leave, but he expects to be back in 30 mins. Shall we discuss sync options between ourselves, in the meantime?

[00:59] shilpa: Sure. I will have to go in am hour.

[00:59] j_k9: That's ok. Let's hope he's back before then :)

[00:59] shilpa: Yes.

[00:59] j_k9: Did you have a chance to read the articles I sent round?

[01:00] shilpa: I read through the mailang links.

[01:00] shilpa: I missed on article.

[01:01] j_k9: Great! Is there any sync design that is suitable for Mira that really stands out, in your opinion?

[01:02] shilpa: I was looking at the article you send.

[01:02] j_k9: Which one?

[01:02] shilpa: So basically we need to sync our database layer

[01:03] shilpa: google/apis/gears

[01:04] j_k9: Hmm, it wouldn't just be our database layer - it would be the part of the directory layer, and all of the Utility data

[01:04] j_k9: I do like the Google Gears article

[01:04] j_k9: http://code.google.com/apis/gears/architecture.html

[01:05] j_k9: I like the idea of a Data Switch, which decides whether to pull data from the internet or from a local data layer depending on whether the computer is online.

[01:05] j_k9: I also like the idea of a sync engine, which keeps that local data layer in sync with the Server's. I'm just wondering how we could create a more detailed design for this!

[01:05] shilpa: Aren't we using database layer to store the directory data too.

[01:06] j_k9: Yes, but remember, the Client shouldn't be aware of which backend the Server's Directory layer is using - it should be irrelevant to the Client whether the Server's Directory layer is using XML files/a database/etc.

[01:07] j_k9: I think our sync system would have to be very flexible, because the data provided by each Utility varies wildly. e.g. Compare the data used by the Files Utility to the sort of data which will be passed by the Discussion Utility.

[01:08] shilpa: Ok. tHANK YOU FOR REMINDING.

[01:08] j_k9: I think the application should be modeless - it would provide the best user experience, and make using Mira a lot simpler IMO.

[01:09] shilpa: Sure.

[01:09] j_k9: However, I think we'd need to offer Utility developers the option of creating a Utility either with manual sync (i.e. the user has to manually choose to push their changes) or with background sync.

[01:09] j_k9: Which complicates matters even further…

[01:10] shilpa: Right.

[01:10] shilpa: So will the sync protocol be the part of directory level?

[01:11] j_k9: I think the sync protocol would be part of the network layer - it would require its own parser. However, if we want to implement this concept of a Data Switch, that would probably occur in the Directory layer.

[01:11] j_k9: Similarly, the Sync Engine would probably be part of the Directory layer.

[01:12] j_k9: For me, the two key design issues to resolve are:

[01:12] j_k9: - How do we support the synchronisation of different types of data? (think: different Utilities, e.g. Workplace Chat Utility and Files Utility)

[01:13] j_k9: - How do we allow Utility developers to choose whether to sync data in the background with high frequency, or require the user to sync manually?

[01:13] j_k9: The first one being the most critical.

[01:14] shilpa: ok

[01:14] j_k9: Because a developer could create an entirely new Utility, such as, say, a chess game - they would need to be able to synchronise this data on-the-fly (i.e. background sync), and this data would probably be unlike the data of any other Utility.

[01:15] j_k9: Or, a better example of a completely radical Utility which might test the flexibility of our sync system: a virtual whiteboard.

[01:15] shilpa: Yes.

[01:17] shilpa: Getting the gravity of it.

[01:17] j_k9: I think we'd have to leave this to the Utility developer, to some extent… i.e. we would provide the sync system, and the Utility developer would configure the sync parameters in order to decide for *themselves* which data will be synchronised, and how.

[01:17] j_k9: That's the only way I see this working? But then again, I have no in-depth experience with C++, let alone a sync system of this complexity!

[01:17] shilpa: Right. Like options auto sync/ manual sync

[01:18] j_k9: Exactly. And not just that:

[01:18] j_k9: Say the Files Utility. That would need to synchronise binary data as revisions, e.g. deltas.

[01:18] j_k9: The Workplace Chat Utility. That would need to synchronise XML data (or other text data)

[01:19] j_k9: The Virtual Whiteboard Utility. That would need to synchronise, hmm.. more complex text data, I suppose?

[01:20] j_k9: Hmm, maybe it's not *that* complicated. Maybe we just need to provide the ability to synchronise text data, and binary data. And then provide the appropriate parameters for the Utility developer to choose which of these systems to use, and which data to use with it.

[01:20] shilpa: One thing we can do is have all the data converted to string. Too early to say but can be an option

[01:20] j_k9: Would that be suitable for binary data?

[01:21] shilpa: I am not sure.

[01:21] j_k9: This is interesting: http://stackoverflow.com/questions/441203/proper-way-to-store-binary-data-with-c-stl

[01:21] shilpa: What I was thinking for this to work we need to have type indepndent system.

[01:22] j_k9: But again, I'm not sure if we'd need to do that (or store binary data in vector<char>, as that article suggests).

[01:22] shilpa: What I mean is encapulate it in some strct/ string at client/server

[01:22] clsk: I'm back

[01:22] j_k9: Just for its transmission over the network to the Client/Server, you mean?

[01:23] shilpa: and the parse it at server

[01:23] shilpa: or client

[01:23] shilpa: yes.

[01:23] j_k9: clsk: That was quick :)

[01:23] j_k9: Shilpa: Right.

[01:23] clsk: I'm going to start reading what you guys have discussed so far. I'll let you know when I catch up

[01:23] clsk: I tried :)

[01:23] shilpa: ok

[01:24] j_k9: When we used the outsourcer last year, Shafi (the developer) worked on writing a wrapper for xdelta. Our intention was to use that for the generation of deltas on binary data.

[01:24] shilpa: ok

[01:24] j_k9: I wonder if we would use that as part of the sync system for binary data? Assuming the sync system is split into two sections: text data, and binary data.

[01:26] shilpa: let's wait for Alan.

[01:27] j_k9: Ok.

[01:27] clsk: hm I've been thinking of something simpler (at least for now)

[01:27] clsk: to let utility developers synch their data

[01:28] clsk: a synchronization system that would work for utility developers is an ambitious project

[01:29] j_k9: In other words, let's not run before we walk :) So how are you suggesting that Utility developer synch their data?

[01:30] clsk: the same way the files utility synchs its data currently

[01:30] clsk: just use regular UT messages

[01:31] shilpa: Ok

[01:31] shilpa: Sound good starting point.

[01:31] clsk: but.. it definitely would be nice to have something like what you guys are discussing

[01:31] clsk: it's just really ambitious. It would take time to design something like this

[01:32] j_k9: Right… So what we're discussing, we should probably leave for something like v0.5 :)

[01:32] clsk: hm I don't know to be honest

[01:32] clsk: perhaps we could give it a try

[01:32] clsk: It would definitely be fun to work on something like that

[01:32] shilpa: Alan how do we address conflict.

[01:33] j_k9: It would just be very complicated. We'd have to decide exactly what to provide to the Utilities, and what the Utilities should handle for themselves

[01:33] clsk: That's a big issue with a generic synch system

[01:34] j_k9: I suppose the Utility developer would have the implement their own conflict resolution, specifically for the Utility?

[01:34] j_k9: *have to

[01:35] shilpa: I think that is redundant.

[01:35] shilpa: But simpler

[01:36] shilpa: Alan how is the file utility taking care of conflicts.

[01:36] clsk: there really aren't conflicts in the file utility

[01:36] clsk: the file utility is just a simple file server

[01:36] clsk: you upload and download files.

[01:36] j_k9: But what happens if we make the Files Utility accessible offline, in this version? Let's say that involves having a local copy of the Files Utility for every Workplace.

[01:37] j_k9: What if you edit a file locally within one of the Files Utility directories?

[01:37] shilpa: an if two member modify the same file

[01:40] clsk: well.. I wasn't thinking that would be possible in the files utility

[01:40] clsk: that's something we would implement later on.. probably on 0.4

[01:42] j_k9: Hmm… Ok. So for this release, what would you like to work on wrt the synch system: allowing a user to log in offline? No features other than logging in and viewing the Workplace?

[01:42] shilpa: Then what should we handle between offline and online state of a user?

[01:43] clsk: be able to see what files exist on the workplace… maybe the ability to queue files to be uploaded when the user comes online

[01:43] clsk: but it would be a fill replacement

[01:43] clsk: full*

[01:44] j_k9: What do you mean by “full replacement”? What would replace what?

[01:45] shilpa: The whole file not delta

[01:45] shilpa: ?

[01:45] clsk: a file being uploaded, would be either a new file; if no file with that name exist. Or would replace an existing file, if another file with that name exist

[01:45] j_k9: Wouldn't the user have to delete the file and then upload their amended version?

[01:45] j_k9: Ah.

[01:45] j_k9: Hmm… Wouldn't the best option be to show a dialog saying “Are you sure you want to overwrite this file” etc.?

[01:46] clsk: that too

[01:46] clsk: I was just talking about the general idea

[01:46] j_k9: Right. Sorry, that was completely off-topic - just a thought :)

[01:46] clsk: it's ok :)

[01:47] j_k9: The first thing that I think needs to be changed is the login system/Directory layer.

[01:47] j_k9: I suppose, after a user connects to a Server for the first time, we would cache their auth details locally so that they could log in to that account offline?

[01:48] j_k9: I also think we should begin the move towards the Manage Accounts system in place of the current login system.

[01:48] clsk: yep

[01:48] shilpa: yE

[01:48] shilpa: Yes

[01:48] clsk: hm.. well that's probably not very safe

[01:49] clsk: If we're going to store the password locally, we need to find a safe way to do so across all 3 OSs we support

[01:49] j_k9: If not, couldn't we store a session, of sorts?

[01:49] clsk: hm what do you mean?

[01:50] j_k9: I'm sure there must be a FOSS app which does this, but I'm thinking of Steam atm. You log in to the app, but you can choose to use the account offline as well.

[01:50] j_k9: Once you have authenticated the first time, it must store some sort of session cache locally - so it wouldn't actually contain any auth data, just data that allows you to access your account via the session.

[01:51] j_k9: I'm not sure how better to explain myself, because I have no idea how Steam works - it's about a proprietary as software comes. :) Just a bit of guesswork.

[01:52] shilpa: Session key sort of thing?

[01:52] j_k9: I guess so

[01:55] j_k9: Ok, so let's break down the targets for the synch system for this release. Are they as follows?

[01:56] j_k9: - Redesign login system (to Manage Accounts) - allow offline login

[01:56] j_k9: - Cache all Utility data to allow for offline viewing

[01:57] j_k9: - Permit queueing of jobs for execution upon reconnection

[01:57] j_k9: I presume the last two would have to be implemented Utility by Utility? i.e. hard-coded into each Utility to support offline use?

[01:58] shilpa: That was good break down.

[01:59] j_k9: What if, for the Files Utility, we maintain a (hidden) copy of the files on the Client? That way, they can be accessed (and “downloaded” (i.e. copied) to another location on the local disk) when the user is offline.

[02:00] j_k9: We could do the same for the Notes Utility. Maintain a cache of all Notes since the last time the user was online.

[02:00] j_k9: But then we get issues like conflict resolution etc. raising their ugly head.

[02:01] shilpa: right.

[02:01] j_k9: Hmm, I'm not sure… clsk, what's your take on this?

[02:02] clsk: hmm.. For the notes utility it makes sense

[02:02] clsk: it's just text

[02:02] clsk: Not sure about the files utility

[02:02] clsk: I've thought about it… but I'm not sure if it's a good idea

[02:03] j_k9: But even the Notes Utility - what if a note is edited offline? Would this amendment be synched when the user is next online? And what if there's a conflict if e.g. another user has edited it in that time?

[02:05] shilpa: I think if we introduce the offline concept we will have to take care of conflict.

[02:08] shilpa: Need to think more on this one.

[02:09] shilpa: i need to go. I won't be able to join in the Monday meeting

[02:09] shilpa: Good night. Bye.

[02:10] shilpa left the chat room.

[02:11] j_k9: I'm a bit late to reply, but bye Shilpa!

[02:11] j_k9: clsk, you still there?

[02:14] clsk left the chat room. (Ping timeout: 258 seconds)

[02:18] badri joined the chat room.

[02:19] j_k9: Hi badri

[02:19] badri: guys am sorry .. i totally forgot we had a meeting today

[02:19] badri: what i missed?

[02:19] clsk joined the chat room.

[02:19] clsk: hm I think I got disconnected

[02:19] j_k9: Hi clsk

[02:19] j_k9: You pinged out, apparently :/

[02:20] j_k9: Badri: We've been discussing the synch system, and what we're going to implement for this release.

[02:20] j_k9: So far, we have the following targets (I'm not sure these are final):

[02:20] j_k9: - Redesign login system (to Manage Accounts) - allow offline login

[02:20] j_k9: - Cache all Utility data to allow for offline viewing

[02:20] j_k9: - Permit queueing of jobs for execution upon reconnection

[02:21] badri: ok

[02:22] badri: i will read tro the archives

[02:22] badri: and see what you decided

[02:24] j_k9: I think we've decided to keep it very simple, for now.

[02:24] j_k9: For example, full synchronisation of the Files Utility (including offline/online) probably won't happen until version 0.4.

[02:25] j_k9: We were also discussing a full-scale generic synchronisation system which Utilities could plug into, but that is an enormous task which would take a lot of work - so we've appreciated the idea, and we've shelved it :P

[02:31] j_k9: Is there anything else we need to discuss about this, or are our synchronisation/offline targets clear enough for this release?

[02:31] clsk: I think they're clear. For now at least

[02:32] j_k9: Ok. In that case, let's call this meeting to a close. I'm sorry it's taken so long - we've really discussed quite different extremes! But at least we'll have something to work from as a base if we ever decide to implement anything more advanced.

[02:32] j_k9: Thank you to both of you (and Shilpa) for joining. :)