TDE core libraries
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

DESIGN 11KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272
  1. DESIGN:
  2. =======
  3. libtdeio uses tdeioslaves (separate processes) that handle a given protocol.
  4. Launching those slaves is taken care of by the tdeinit/tdelauncher tandem,
  5. which are notified by DCOP.
  6. Connection is the most low-level class, the one that encapsulates the pipe.
  7. SlaveInterface is the main class for transferring anything to the slave
  8. and Slave, which inherits SlaveInterface, is the sub class that Job should handle.
  9. A slave inherits SlaveBase, which is the other half of SlaveInterface.
  10. The scheduling is supposed to be on a two level basis. One is in the daemon
  11. and one is in the application. The daemon one (as opposite to the holy one? :)
  12. will determine how many slaves are ok for this app to be opened and it will
  13. also assign tasks to actually existing slaves.
  14. The application will still have some kind of a scheduler, but it should be
  15. a lot simpler as it doesn't have to decide anything besides which
  16. task goes to which pool of slaves (related to the protocol/host/user/port)
  17. and move tasks around.
  18. Currently a design study to name it cool is in scheduler.cpp but in the
  19. application side. This is just to test other things like recursive jobs
  20. and signals/slots within SlaveInterface. If someone feels brave, the scheduler
  21. is yours!
  22. On a second thought: at the daemon side there is no real scheduler, but a
  23. pool of slaves. So what we need is some kind of load calculation of the
  24. scheduler in the application and load balancing in the daemon.
  25. A third thought: Maybe the daemon can just take care of a number of 'unused'
  26. slaves. When an application needs a slave, it can request it from the daemon.
  27. The application will get one, either from the pool of unused slaves,
  28. or a new one will be created. This keeps things simple at the daemon level.
  29. It is up to the application to give the slaves back to the daemon.
  30. The scheduler in the application must take care not to request too many
  31. slaves and could implement priorities.
  32. Thought on usage:
  33. * Typically a single slave-type is used exclusively in one application. E.g.
  34. http slaves are used in a web-browser. POP3 slaves used in a mail program.
  35. * Sometimes a single program can have multiple roles. E.g. konqueror is
  36. both a web-browser and a file-manager. As a web-browser it primarily uses
  37. http-slaves as a file-manager file-slaves.
  38. * Selecting a link in konqueror: konqueror does a partial download of
  39. the file to check the mimetype (right??) then the application is
  40. started which downloads the complete file. In this case it should
  41. be able to pass the slave which does the partial download from konqueror
  42. to the application where it can do the complete download.
  43. Do we need to have a hard limit on the number of slaves/host?
  44. It seems so, because some protocols are about to fail if you
  45. have two slaves running in parralel (e.g. POP3)
  46. This has to be implemented in the daemon because only at daemon
  47. level all the slaves are known. As a consequence slaves must
  48. be returned to the daemon before connecting to another host.
  49. (Returning the slaves back to the daemon after every job is not
  50. strictly needed and only causes extra overhead)
  51. Instead of actually returning the slave to the daemon, it could
  52. be enough to ask 'recycling permission' from the daemon: the
  53. application asks the daemon whether it is ok to use a slave for
  54. another host. The daemon can then update its administration of
  55. which slave is connected to which host.
  56. The above does of course not apply to hostless protocols (like file).
  57. (They will never change host).
  58. Apart from a 'hard limit' on the number of slaves/host we can have
  59. a 'soft limit'. E.g. upon connection to a HTTP 1.1 server, the web-
  60. server tells the slave the number of parallel connections allowed.
  61. THe simplest solution seems to be to treat 'soft limits' the same
  62. as 'hard limits'. This means that the slave has to communicate the
  63. 'soft limit' to the daemon.
  64. Jobs using multiple slaves.
  65. If a job needs multiple slaves in parallel (e.g. copying a file from
  66. a web-server to a ftp-server or browsing a tar-file on a ftp-site)
  67. we must make sure to request the daemon for all slaves together since
  68. otherwise there is a risk of deadlock.
  69. (If two applications both need a 'pop3' and a 'ftp' slave for a single
  70. job and only a single slave/host is allowed for pop3 and ftp, we must
  71. prevent giving the single pop3 slave to application #1 and the single
  72. ftp slave to application #2. Both applications will then wait till the
  73. end of times till they get the other slave so that they can start the
  74. job. (This is a quite unlikely situation, but nevertheless possible))
  75. File Operations:
  76. listRecursive is implemented as listDir and finding out if in the result
  77. is a directory. If there is, another listDir job is issued. As listDir
  78. is a readonly operation it fails when a directory isn't readable
  79. .. but the main job goes on and discards the error, because
  80. bIgnoreSubJobsError is true, which is what we want (David)
  81. del is implemented as listRecursive, removing all files and removing all
  82. empty directories. This basically means if one directory isn't readable
  83. we don't remove it as listRecursive didn't find it. But the del will later
  84. on try to remove it's parent directory and fail. But there are cases when
  85. it would be possible to delete the dir in chmod the dir before. On the
  86. other hand del("/") shouldn't list the whole file system and remove all
  87. user owned files just to find out it can't remove everything else (this
  88. basically means we have to take care of things we can remove before we try)
  89. ... Well, rm -rf / refuses to do anything, so we should just do the same:
  90. use a listRecursive with bIgnoreSubJobsError = false. If anything can't
  91. be removed, we just abort. (David)
  92. ... My concern was more that the fact we can list / doesn't mean we can
  93. remove it. So we shouldn't remove everything we could list without checking
  94. we can. But then the question arises how do we check whether we can remove it?
  95. (Stephan)
  96. ... I was wrong, rm -rf /, even as a user, lists everything and removes
  97. everything it can (don't try this at home!). I don't think we can do
  98. better, unless we add a protocol-dependent "canDelete(path)", which is
  99. _really_ not easy to implement, whatever protocol. (David)
  100. Lib docu
  101. ========
  102. mkdir: ...
  103. rmdir: ...
  104. chmod: ...
  105. special: ...
  106. stat: ...
  107. get is implemented as TransferJob. Clients get 'data' signals with the data.
  108. A data block of zero size indicates end of data (EOD)
  109. put is implemented as TransferJob. Clients have to connect to the
  110. 'dataReq' signal. The slave will call you when it needs your data.
  111. mimetype: ...
  112. file_copy: copies a single file, either using CMD_COPY if the slave
  113. supports that or get & put otherwise.
  114. file_move: moves a single file, either using CMD_RENAME if the slave
  115. supports that, CMD_COPY + del otherwise, or eventually
  116. get & put & del.
  117. file_delete: delete a single file.
  118. copy: copies a file or directory, recursively if the latter
  119. move: moves a file or directory, recursively if the latter
  120. del: deletes a file or directory, recursively if the latter
  121. PROGRESS DISPLAYING :
  122. =====================
  123. Taj brought up the idea of deligating all progress informations to an extern
  124. GUI daemon which could be provided in several implementations - examples
  125. are popup dialogs (most are annoyed by them, like me :) or a kicker applet
  126. or something completely different. This would also remove the dependency on
  127. libtdeui (I hope).
  128. Conclusion: tdeio_uiserver is this single GUI daemon, but the dependency on
  129. libtdeui couldn't be removed (for many reasons, including Job::showErrorDialog())
  130. A. progress handling
  131. ---------------------
  132. There will be two ways how the application can display progress :
  133. 1. regular apps will use NetAccess for all tdeio operations and will not care
  134. about progress handling :
  135. - NetAccess creates Job
  136. - NetAccess creates JobObserver that will connect to the Job's signals and
  137. pass them via dcop to the running GUI Progress Server
  138. 2. apps that want to do some handling with progress dialogs like Caitoo or
  139. KMail :
  140. - app creates Job
  141. - app creates a progress dialog : this should be a ProgressBase descendant
  142. e.g. StatusProgress or custom progress dialog
  143. - app calls progress->setJob( job ) in order to connect job's signals with
  144. progress dialog slots
  145. B. customized progress dialogs
  146. -------------------------------
  147. This will be similar to what we had before.
  148. - ProgressBase class that all other dialogs will inherit.
  149. will contain an initialization method setJob( TDEIO::Job*) for apps of the
  150. second class (see A.2 above), that will connect job's signals to dialog's
  151. slots
  152. - DefaultProgress ( former KIOSimpleProgressDialog ) that will be used for
  153. regular progress dialogs created by GUI Progress Server
  154. - StatusProgress ( former KIOLittleProgressDialog ) that can be used for
  155. embedding in status bar
  156. C. GUI Progress Server
  157. -----------------------
  158. This is a special progress server.
  159. - createProgress() will either create a DefaultProgress dialog or add new entry
  160. in a ListProgress ( an all-jobs-in-one progress dialog )
  161. - after receiving signals from the JobObserver via DCOP it will call
  162. appropriate method of progress dialog ( either in DefaultProgress or ListProgress )
  163. - ListProgres can be a Caitoo style dialog, kicker applet or both in one.
  164. D. Some notes
  165. --------------
  166. 1. most of the apps will not care at all about the progress display
  167. 2. user will be able to choose whether he wants to see separate progress
  168. dialogs or all-in-one ListProgress dialog
  169. 3. developers can create their custom progress dialogs that inherit
  170. ProgressBase and do any manipulation with a dialog if they use a second
  171. approach ( see A.2 above )
  172. Streaming
  173. ---------
  174. 1. We currently support a streaming "GET": e.g. file:/tmp/test.gz#gzip:/
  175. works. The following should also work: file:/tmp/test.gz.gz#gzip:/#gzip:/
  176. The current approach makes a TrasnferJob for gzip:/ and then adds a
  177. subjob for "file:/tmp/test.gz.gz#gzip:/" which itself adds a subjob
  178. for "file:/tmp/test.gz.gz".
  179. 2. This doesn't extend very well to PUT, because there the order should
  180. basically be the other way around, but the "input" to the job as a whole
  181. should go to the "gzip:/" job, not to the "file:/tmp/test.gz.gz."
  182. It would probably be easier to implement such a job in the way the
  183. current "CopyJob" is done. Have a Job and make all sub-urls sub-jobs of
  184. this Job.
  185. 3. As a result of 1. COPY FROM an url like file:/tmp/test.gz#gzip:/ should
  186. work. COPY TO does not, because that would require PUT.
  187. Resuming
  188. --------
  189. A rough note for now, just to have this somewhere :
  190. (PJ=put-job, GJ=get-job)
  191. PJ can't resume:
  192. PJ-->app: canResume(0) (emitted by dataReq)
  193. GJ-->app: data()
  194. PJ-->app: dataReq()
  195. app->PJ: data()
  196. PJ can resume but GJ can't resume:
  197. PJ-->app: canResume(xx)
  198. app->GJ: start job with "resume=xxx" metadata.
  199. GJ-->app: data()
  200. PJ-->app: dataReq()
  201. app->PJ: data()
  202. PJ can resume and GJ can resume:
  203. PJ-->app: canResume(xx)
  204. app->GJ: start job with "resume=xxx" metadata.
  205. GJ-->app: canResume(xx)
  206. GJ-->app: data()
  207. PJ-->app: dataReq()
  208. app->PJ: canResume(xx)
  209. app->PJ: data()
  210. So when the slave supports resume for "put" it has to check after the first
  211. dataRequest() whether it has got a canResume() back from the app. If it did
  212. it must resume. Otherwise it must start from 0.