![]() |
| Home > FAQ on different themes > |
AFS distributed filesystem FAQ |
Section 3 of 4 - Prev - Next
All sections - 1 - 2 - 3 - 4
system. At the lowest level is the UDP protocol, which is part of TCP/IP. UDP is the connection to the actual network wire. The next protocol level is the remote procedure call (RPC). In general, RPCs allow the developer to build applications using the client/server model, hiding the underlying networking mechanisms. AFS uses Rx, an RPC protocol developed specifically for AFS during its development phase at Carnegie Mellon University. Above the RPC is a series of server processes and interfaces that all use Rx for communication between machines. Fileserver, volserver, upserver, upclient, and bosserver are server processes that export RPC interfaces to allow their user interface commands to request actions and get information. For example, a bos statuscommand will examine the bos server process on the indicated file server machine. Database servers use ubik, a replicated database mechanism which is implemented using RPC. Ubik guarantees that the copies of AFS databases of multiple server machines remain consistent. It provides an application programming interface (API) for database reads and writes, and uses RPCs to keep the database synchronized. The database server processes, vlserver, kaserver, and ptserver, reside above ubik. These processes export an RPC interface which allows user commands to control their operation. For instance, the pts command is used to communicate with the ptserver, while the command klog uses the kaserver's RPC interface. Some application programs are quite complex, and draw on RPC interfaces for communication with an assortment of processes. Scout utilizes the RPC interface to file server processes to display and monitor the status of file servers. The uss command interfaces with kaserver, ptserver, volserver and vlserver to create new user accounts. The Cache Manager also exports an RPC interface. This interface is used principally by file server machines to break callbacks. It can also be used to obtain Cache Manager status information. The program cmdebug shows the status of a Cache Manager using this interface. For additional information, Section 1.5 of the AFS System Administrator's Guide and the April 1990 Cache Update contain more information on ubik. Udebug information and short descriptions of all debugging tools were included in the January 1991 Cache Update. Future issues will discuss other debugging tools in more detail. [ source: ftp://ftp.transarc.com/pub/afsug/newsletter/apr91 ] [ Copyright 1991 Transarc Corporation ] Subject: 3.06 Are setuid programs executable across AFS cell boundaries? By default, the setuid bit is ignored but the program may be run (without setuid privilege). It is possible to configure an AFS client to honour the setuid bit. This is achieved by root running: root@toontown # fs setcell -cell $cellname -suid (where $cellname is the name of the foreign cell. Use with care!). NB: making a program setuid (or setgid) in AFS does *not* mean that the program will get AFS permissions of a user or group. To become AFS authenticated, you have to klog. If you are not authenticated, AFS treats you as "system:anyuser". Subject: 3.07 How does AFS maintain consistency on read-write files? AFS uses a mechanism called "callback". Callback is a promise from the fileserver that the cache version of a file/directory is up-to-date. It is established by the fileserver with the caching of a file. When a file is modified the fileserver breaks the callback. When the user accesses the file again the Cache Manager fetches a new copy if the callback has been broken. The following paragraphs describe AFS callback mechanism in more detail: If I open() fileA and start reading, and you then open() fileA, write() a change ***and close() or fsync()*** the file to get your changes back to the server - at the time the server accepts and writes your changes to the appropriate location on the server disk, the server also breaks callbacks to all clients to which it issued a copy of fileA. So my client receives a message to break the callback on fileA, which it dutifully does. But my application (editor, spreadsheet, whatever I'm using to read fileA) is still running, and doesn't really care that the callback has been broken. When something causes the application to read() more of the file the read() system call executes AFS cache manager code via the VFS switch, which does check the callback and therefore gets new copies of the data. Of course, the application may not re-read data that it has already read, but that would also be the case if you were both using the same host. So, for both AFS and local files, I may not see your changes. Now if I exit the application and start it again, or if the application does another open() on the file, then I will see the changes you've made. This information tends to cause tremendous heartache and discontent - but unnecessarily so. People imagine rampant synchronization problems. In practice this rarely happens and in those rare instances, the data in question is typically not critical enough to cause real problems or crashing and burning of applications. Since 1985, we've found that the synchronization algorithm has been more than adequate in practice - but people still like to worry! The source of worry is that, if I make changes to a file from my workstation, your workstation is not guaranteed to be notified until I close or fsync the file, at which point AFS guarantees that your workstation will be notified. This is a significant departure from NFS, in which no guarantees are provided. Partially because of the worry factor and largely because of Posix, this will change in DFS. DFS synchronization semantics are identical to local file system synchronization. [ DFS is the Distributed File System which is part of the Distributed ] [ Computing Environment (DCE). ] Subject: 3.08 How can I run daemons with tokens that do not expire? It is not a good idea to run with tokens that do not expire because this would weaken one of the security features of Kerberos. A better approach is to re-authenticate just before the token expires. There are two examples of this that have been contributed to afs-contrib. The first is "reauth": file:///afs/transarc.com/public/afs-contrib/tools/reauth/ ftp://ftp.transarc.com/pub/afs-contrib/tools/reauth/ The second is "lat": /afs/transarc.com/public/afs-contrib/pointers\ /UMich-lat-authenticated-batch-jobs ftp://ftp.transarc.com/pub/afs-contrib/pointers /UMich-lat-authenticated-batch-jobs Subject: 3.09 Can I check my user's passwords for security purposes? Yes. Alec Muffett's Crack tool (at version 4.1f) has been converted to work on the Transarc kaserver database. This modified Crack (AFS Crack) is available via anonymous ftp from: ftp://export.acs.cmu.edu/pub/crack.tar.Z and is known to work on: pmax_* sun4*_* hp700_* rs_aix* next_* It uses the file /usr/afs/db/kaserver.DB0, which is the database on the kaserver machine that contains the encrypted passwords. As a bonus, AFS Crack is usually two to three orders of magnitude faster than the standard Crack since there is no concept of salting in a Kerberos database. On a normal UNIX /etc/passwd file, each password can have been encrypted around 4096 (2^12) different saltings of the crypt(3) algorithm, so for a large number of users it is easy to see that a potentially large (up to 4095) number of seperate encryptions of each word checked has been avoided. Author: Dan Lovinger Contact: Derrick J. Brashear Note: AFS Crack does not work for MIT Kerberos Databases. The author is willing to give general guidance to someone interested in doing the (probably minimal) amount of work to port it to do MIT Kerberos. The author does not have access to a MIT Kerberos server to do this. Subject: 3.10 Is there a way to automatically balance disk usage across fileservers? Yes. There is a tool, balance, which does exactly this. It can be retrieved via anonymous ftp from: ftp://ftp.andrew.cmu.edu/pub/balance-1.1a.tar.Z Actually, it is possible to write arbitrary balancing algorithms for this tool. The default set of "agents" provided for the current version of balance balance by usage, # of volumes, and activity per week, the latter currently requiring a source patch to the AFS volserver. Balance is highly configurable. Author: Dan Lovinger Contact: Derrick Brashear Subject: 3.11 Can I shutdown an AFS fileserver without affecting users? Yes, this is an example of the flexibility you have in managing AFS. Before attempting to shutdown an AFS fileserver you have to make some arrangements that any services that were being provided are moved to another AFS fileserver: 1) Move all AFS volumes to another fileserver. (Check you have the space!) This can be done "live" while users are actively using files in those volumes with no detrimental effects. 2) Make sure that critical services have been replicated on one (or more) other fileserver(s). Such services include: kaserver - Kerberos Authentication server vlserver - Volume Location server ptserver - Protection server buserver - Backup server It is simple to test this before the real shutdown by issuing: bos shutdown $server $service where: $server is the name of the server to be shutdown and $service is one (or all) of: kaserver vlserver ptserver buserver Other points to bear in mind: + "vos remove" any RO volumes on the server to be shutdown. Create corresponding RO volumes on the 2nd fileserver after moving the RW. There are two reasons for this: 1) An RO on the same partition ("cheap replica") requires less space than a full-copy RO. 2) Because AFS always accesses RO volumes in preference to RW, traffic will be directed to the RO and therefore quiesce the load on the fileserver to be shutdown. + If the system to be shutdown has the lowest IP address there may be a brief delay in authenticating because of timeout experienced before contacting a second kaserver. Subject: 3.12 How can I set up mail delivery to users with $HOMEs in AFS? There are many ways to do this. Here, only two methods are considered: Method 1: deliver into local filestore This is the simplest to implement. Set up your mail delivery to append mail to /var/spool/mail/$USER on one mailserver host. The mailserver is an AFS client so users draw their mail out of local filestore into their AFS $HOME (eg: inc). Note that if you expect your (AFS unauthenticated) mail delivery program to be able to process .forward files in AFS $HOMEs then you need to add "system:anyuser rl" to each $HOMEs ACL. The advantages are: + Simple to implement and maintain. + No need to authenticate into AFS. The drawbacks are: - It doesn't scale very well. - Users have to login to the mailserver to access their new mail. - Probably less secure than having your mailbox in AFS. - System administrator has to manage space in /var/spool/mail. Method 2: deliver into AFS This takes a little more setting up than the first method. First, you must have your mail delivery daemon AFS authenticated (probably as "postman"). The reauth example in afs-contrib shows how a daemon can renew its token. You will also need to setup the daemon startup soon after boot time to klog (see the -pipe option). Second, you need to set up the ACLs so that "postman" has lookup rights down to the user's $HOME and "lik" on $HOME/Mail. Advantages: + Scales better than first method. + Delivers to user's $HOME in AFS giving location independence. + Probably more secure than first method. + User responsible for space used by mail. Disadvantages: - More complicated to set up. - Need to correctly set ACLs down to $HOME/Mail for every user. - Probably need to store postman's password in a file so that the mail delivery daemon can klog after boot time. This may be OK if the daemon runs on a relatively secure host. An example of how to do this for IBM RISC System/6000 is auth-sendmail. A beta test version of auth-sendmail can be found in: file:///afs/transarc.com/public/afs-contrib/doc/faq/auth-sendmail.tar.Z ftp://ftp.transarc.com/pub/afs-contrib/doc/faq/auth-sendmail.tar.Z Subject: 3.13 Should I replicate a ReadOnly volume on the same partition and server as the ReadWrite volume? Yes, Absolutely! It improves the robustness of your served volumes. If ReadOnly volumes exist (note use of term *exist* rather than *are available*), Cache Managers will *never* utilize the ReadWrite version of the volume. The only way to access the RW volume is via the "dot" path (or by special mounting). This means if *all* RO copies are on dead servers, are offline, are behind a network partition, etc, then clients will not be able to get the data, even if the RW version of the volume is healthy, on a healthy server and in a healthy network. However, you are *very* strongly encouraged to keep one RO copy of a volume on the *same server and partition* as the RW. There are two reasons for this: 1) The RO that is on the same server and partition as the RW is a clone (just a copy of the header - not a full copy of each file). It therefore is very small, but provides access to the same set of files that all other (full copy) ReadOnly volume do. Transarc trainers refer to this as the "cheap replica". 2) To prevent the frustration that occurs when all your ROs are unavailable but a perfectly healthy RW was accessible but not used. If you keep a "cheap replica", then by definition, if the RW is available, one of the RO's is also available, and clients will utilize that site. Subject: 3.14 Should I start AFS before NFS in /etc/inittab? Yes, it is possible to run both AFS and NFS on the same system but you should start AFS first. In IBM's AIX 3.2, your /etc/inittab would contain: rcafs:2:wait:/etc/rc.afs > /dev/console 2>&1 # Start AFS daemons rcnfs:2:wait:/etc/rc.nfs > /dev/console 2>&1 # Start NFS daemons With AIX, you need to load NFS kernel extensions before the AFS KEs in /etc/rc.afs like this: #!/bin/sh - # example /etc/rc.afs for an AFS fileserver running AIX 3.2 # echo "Installing NFS kernel extensions (for AFS+NFS)" /etc/gfsinstall -a /usr/lib/drivers/nfs.ext echo "Installing AFS kernel extensions..." D=/usr/afs/bin/dkload ${D}/cfgexport -a ${D}/export.ext ${D}/cfgafs -a ${D}/afs.ext /usr/afs/bin/bosserver & Subject: 3.15 Will AFS run on a multi-homed fileserver? (multi-homed = host has more than one network interface.) Yes, it will. However, AFS was designed for hosts with a single IP address. There can be problems if you have one host name being resolved to several IP addresses. Transarc suggest designating unique hostnames for each network interface. For example, a host called "spot" has two tokenring and one ethernet interfaces: spot-tr0, spot-tr1, spot-en0. Then, select which interface will be used for AFS and use that hostname in the CellServDB file (eg: spot-tr0). You also have to remember to use the AFS interface name with any AFS commands that require a server name (eg: vos listvol spot-tr0). There is a more detailed discussion of this in the August 1993 issue of "Cache Update" (see: ftp://ftp.transarc.com/pub/afsug/newsletter/aug93). The simplest way of dealing with this is to make your AFS fileservers single-homed (eg only use one network interface). At release 3.4 of AFS, it is possible to have multi-homed fileservers (but _not_ multi-homed database servers). Subject: 3.16 Can I replicate my user's home directory AFS volumes? No. Users with $HOMEs in /afs normally have an AFS ReadWrite volume mounted in their home directory. You can replicate a RW volume but only as a ReadOnly volume and there can only be one instance of a ReadWrite volume. In theory, you could have RO copies of a user's RW volume on a second server but in practice this won't work for the following reasons: a) AFS has built-in bias to always access the RO copy of a RW volume. So the user would have a ReadOnly $HOME which is not too useful! b) Even if a) was not true you would have to arrange frequent synchronisation of the RO copy with the RW volume (for example: "vos release user.fred; fs checkv") and this would have to be done for all such user volumes. c) Presumably, the idea of replicating is to recover the $HOME in the event of a server crash. Even if a) and b) were not problems consider what you might have to do to recover a $HOME: 1) Create a new RW volume for the user on the second server (perhaps named "user.fred.2"). 2) Now, where do you mount it? The existing mountpoint cannot be used because it already has the ReadOnly copy of the original volume mounted there. Let's choose: /afs/MyCell/user/fred.2 3) Copy data from the RO of the original into the new RW volume user.fred.2 4) Change the user's entry in the password file for the new $HOME: /afs/MyCell/user/fred.2 You would have to attempt steps 1 to 4 for every user who had their RW volume on the crashed server. By the time you had done all of this, the crashed server would probably have rebooted. The bottom line is: you cannot replicate $HOMEs across servers. Subject: 3.17 Which TCP/IP ports and protocols do I need to enable in order to operate AFS through my Internet firewall? Assuming you have already taken care of nameserving, you may wish to use an Internet timeserver for Network Time Protocol [35] [36]: ntp 123/tcp A list of NTP servers is available via anonymous FTP from: http://www.eecis.udel.edu/~mills/ntp/servers.html For further details on NTP see: http://www.eecis.udel.edu/~ntp/ For a "minimal" AFS service which does not allow inbound or outbound klog: fileserver 7000/udp cachemanager 7001/udp ptserver 7002/udp vlserver 7003/udp kaserver 7004/udp volserver 7005/udp reserved 7006/udp bosserver 7007/udp (Ports in the 7020-7029 range are used by the AFS backup system, and won't be needed by external clients performing simple file accesses.) Additionally, for "klog" to work through the firewall you need to allow inbound and outbound UDP on ports >1024 (probably 1024 . Subject: 3.20 How can I list which clients have cached files from a server? By using the following script: #!/bin/ksh - # # NAME afsclients # AUTHOR Rainer Toebbicke # DATE June 1994 # PURPOSE Display AFS clients which have grabbed files from a server if [ $# = 0 ]; then echo "Usage: $0 ... " exit 1 fi for n; do /usr/afsws/etc/rxdebug -servers $n -allconn done | grep '^Connection' | \ while read x y z ipaddr rest; do echo $ipaddr; done | sort -u | while read ipaddr; do ipaddr=${ipaddr%%,} n="`nslookup $ipaddr`" n="${n##*Name: }" n="${n%%Address:*}" n="${n##*([ ])}" n="${n%?}" echo "$n ($ipaddr)" done Subject: 3.21 Do Backup volumes require as much space as ReadWrite volumes? No. The technique used is to create a new volume, where every file in the RW copy is pointed to by the new backup volume. The files don't exist in the BK, only in the RW volume. The backup volume therefore takes up very little space. If the user now starts modifying data, the old copy must not be destroyed. There is a Copy-On-Write bit in the vnode - if the fileserver writes to a vnode with the bit on it allocates a new vnode for the data and turns off the COW bit. The BK volume hangs onto the old data, and the RW volume slowly splits itself away over time. The BK volume is re-synchronised with the RW next time a "vos backupsys" is run. The space needed for the BK volume is directly related to the size of all files changed in the RW between runs of "vos backupsys". Subject: 3.22 Should I run timed on my AFS client? No. The AFS Cache Manager makes use of NTP [35] [36] to synchronise time with your cell's NTP servers. Typically, one of your AFS cell's servers synchronises with an external NTP server and provides accurate time to your cell. Subject: 3.23 Why should I keep /usr/vice/etc/CellServDB current? On AFS clients, /usr/vice/etc/CellservDB, defines the cells and (their servers) that can be accessed via /afs. Over time, site details change: servers are added/removed or moved onto new network addresses. New sites appear. In order to keep up-to-date with such changes, the CellservDB file on each AFS client should be kept consistent with some master copy (at your site). As well as updating CellservDB, your AFS administrator should ensure that new cells are mounted in your cell's root.afs volume. Subject: 3.24 How can I keep /usr/vice/etc/CellServDB current? Do a daily copy from a master source and update the AFS kernel sitelist. The client CellServDB file must not reside under /afs and is best located in local filespace. Simply updating a client CellServDB file is not enough. You also need to update the AFS kernel sitelist by either: a) rebooting the client or b) running "fs newcell $cell_name $server_list" for each site in the CellServDB file. A script to update the AFS kernel sitelist on a running system is newCellServDB. file:///afs/ece.cmu.edu/usr/awk/Public/newCellServDB ftp://ftp.ece.cmu.edu/pub/afs-tools/newCellServDB One way to distribute CellServDB is to have a root cron job on each AFS client copy the file then run newCellServDB. Example: #!/bin/ksh - # # NAME syncCellServDB # PURPOSE Update local CellServDB file and update AFS kernel sitelist # USAGE run by daily root cron job eg: # 0 3 * * * /usr/local/sbin/syncCellServDB # # NOTE "@cell" is a symbolic link to /afs/$this_cell_name src=/afs/@cell/service/etc/CellServDB dst=/usr/vice/etc/CellServDB xec=/usr/local/sbin/newCellServDB log=/var/log/syncCellServDB if [ -s ${src} ]; then if [ ${src} -nt ${dst} ]; then cp $dst ${dst}- && cp $src $dst && $xec 2>&1 >$log else echo "master copy no newer: no processing to be done" >$log fi else echo "zero length file: ${src}" >&2 fi Subject: 3.25 How can I compute a list of AFS fileservers? Here is a Korn shell command to do it: stimpy@nick $ vos listvldb -cell $(cat /usr/vice/etc/ThisCell) \ | awk '(/server/) {print $2}' | sort -u Subject: 3.26 How can I set up anonymous FTP login to access /afs? The easiest way on a primarily "normal" machine (where you don't want to have everything in AFS) is to actually mount root.cell under ~ftp, and then symlink /afs to ~ftp/afs or whatever. It's as simple as changing the mountpoint in /usr/vice/etc/cacheinfo and restarting afsd. Note that when you do this, anon ftp users can go anywhere system:anyuser can (or worse, if you're using IP-based ACLs and the ftp host is PTS groups). The only "polite" solution I've arrived at is to have the ftp host machine run a minimal CellServDB and police my ACLs tightly. Alternatively, you can make ~ftp an AFS volume and just mount whatever you need under that - this works well if you can keep everything in AFS, and you don't have the same problems with anonymous "escapes" into /afs. Unless you need to do authenticating ftp, you are _strongly_ recommended using wu-ftpdv2.4 (or better). Subject: 3.27 Where can I find the Andrew Benchmark? file:///afs/transarc.com/public/afs-contrib/doc/faq/ab.tar.Z [156k] ftp://ftp.transarc.com/pub/afs-contrib/doc/faq/ab.tar.Z [156k] This is a tar archive of file:///afs/cs.cmu.edu/user/satya/ftp/ab/ Subject: 4 Getting more information Subject: 4.01 Is there an anonymous FTP site with AFS information? Yes, it is: ftp.transarc.com A brief summary of contents: Directory Contents pub/afsug/newsletter AFS user group newsletters pub/afs-contrib Contributed tools and documents pub/afsps/doc release notes, SUPPORTED_SYSTEMS.afs.* pub/afsug AFS user group (see README for detail) pub/afsps/progint AFS programming interface docs These directories are also accessible via AFS. For example: /afs/transarc.com/public/afs-contrib (NB "pub" => "public" when using AFS to access these.) Subject: 4.02 Which USENET newsgroups discuss AFS? alt.filesystems.afs and occasionally in comp.unix.admin. Subject: 4.03 Where can I get training in AFS? Transarc provide user and administrator courses. These can be provided at the customer site or at Transarc's offices. Transarc's education coordinator may be contacted by: telephone: +1 412 338 4363 email: education@transarc.com U http://www.transarc.com Subject: 4.04 Where can I find AFS resources in World Wide Web (WWW)? Here are some I have found (please let me know if you find more): a) A collection of AFS information maintained by Derrick Brashear at CMU: http://www.cs.cmu.edu/afs/andrew.cmu.edu/usr/shadow/www/afs.html (Also accessible in: /afs/andrew.cmu.edu/usr/shadow/www) b) AFS Beginners Guide (ALW/NIH): http://www.alw.nih.gov/Docs/AFS/AFS_toc.html c) NCSA AFS User Guide: http://www.ncsa.uiuc.edu/Pubs/UserGuides/AFSGuide/AFSv2.1Book.html d) Transarc AFS Product Information: U http://www.transarc.com/dfs/public/www/htdocs/.hosts/external/Product/EFS/AFS/afsoverview.html e) CERN AFS User's Guide: http://wsspinfo.cern.ch/file/doc/afsug.html f) MIT SIPB's Inessential AFS http://web.mit.edu/afs/sipb.mit.edu/project/doc/afs/html/afs-new.html g) Stanford University hypermail archive of info-afs@transarc.com http://www-archive.stanford.edu/lists/info-afs.html N h) Linux AFS FAQ: N http://www.umlug.umd.edu/linuxafs/ Subject: 4.05 Is there a mailing list for AFS topics? Yes, it is info-afs@transarc.com. An automated program called Majordomo is now handling the info-afs list. To join the mailing list, send a message to: majordomo@transarc.com In the body (not the Subject line) of the message, type: subscribe info-afs For example: $ mail -s "subscribe to info-afs" majordomo@transarc.com <<% subscribe info-afs % To unsubscribe: $ mail -s "unsubscribe from info-afs" majordomo@transarc.com <<% unsubscribe info-afs Subject: 4.06 Where can I find an archive of info-afs@transarc.com? There is a hypermail archive at: http://www-archive.stanford.edu/lists/info-afs.html Subject: 4.07 Where can I find an archive of alt.filesystems.afs? file:///afs/ibm.uk/common/archive/alt.filesystems.afs/ Both the info-afs@transarc.com and alt.filesystems.afs archives are incomplete. If you have material to contribute, please let me know. Subject: 4.08 Where can I find AFS related GIFs? U file:///afs/transarc.com/public/afs-contrib/doc/faq/images/index.html U ftp://ftp.transarc.com/pub/afs-contrib/doc/faq/images/index.html Subject: 4.09 Gibt es eine deutsche AFS Benutzer Gruppe? Ja, wenn Sie mitmachen wollen, schicken Sie bitte eine E-Mail an: afsdeu-request@hrz.th-darmstadt.de Ueber diese Adresse werden "subscribe" und "unsubscribe" Requests bearbeitet. Subject: 4.10 Donde puedo encontrar informacion en Espanol sobre AFS? Hay algunas notas en Espanol sobre AFS en: http://w3.ing.puc.cl/~cet/afs.html Subject: 5 About the AFS faq I started compiling the FAQ after attending an AFS administrators class and while waiting for the distribution tape to arrive from Transarc (back in July 93). The initial goal was to assist users at my site to understand AFS issues. The FAQ seemed to be a more widely useful resource so it was made generally available. I hope you have found the AFS FAQ useful. Your criticism or suggestions for improving it are welcome, so please don't hesitate to email your views (or just say "hello"). This compilation is dedicated to my AFS teacher and all those who inspire through good humour, enthusiasm, wit and wisdom. -- paul http://acm.org/~mpb/homepage.html Subject: 5.01 How can I get a copy of the AFS faq? If you do make a copy, please be aware that this compilation changes over time: you will need to do a periodic re-copy to keep your copy up-to-date. There are two reference sources: 1) The text only version, available via AFS from: /afs/transarc.com/public/afs-contrib/doc/faq/afs.faq 2) The World Wide Web (HTML) version, available via URL:
Section 3 of 4 - Prev - Next
All sections - 1 - 2 - 3 - 4
| Back to category FAQ on different themes - Use Smart Search |
| Home - Smart Search - About the project - Feedback |
© allanswers.org | Terms of use