iSCSI SCST target driver
========================

Version 3.3.0, XX XXXXX 2016
----------------------------

ISCSI-SCST is a deeply reworked fork of iSCSI Enterprise Target (IET)
(http://iscsitarget.sourceforge.net). Reasons of the fork were:

 - To be able to use full power of SCST core.

 - To fix all the problems, corner cases issues and iSCSI standard
   violations which IET has.

See for more info http://iscsi-scst.sourceforge.net.

This version is compatible with SCST version 2.0.0 and higher.


Installation if your Linux kernel already has iSCSI-SCST built-in
-----------------------------------------------------------------

Simply run "make all", then "make install".


Installation out of Linux kernel tree
-------------------------------------

See HOWTOs in the doc/ subdirectory.

Only vanilla kernels from kernel.org and RHEL/CentOS 5.2 kernels are
supported, but it should work on other (vendors') kernels, if you manage
to successfully compile on them. The main problem with vendor's kernels
is that they often contain patches, which appear only in the next
version of the vanilla kernel, therefore it's quite hard to track such
changes. Thus, if during compilation for some vendor's kernel your
compiler complains about redefinition of some symbol, you should either
switch to vanilla kernel, or add or change as necessary the
corresponding to that symbol "#if LINUX_VERSION_CODE" statement.

Default sysfs interface supports only kernels 2.6.26 and higher, because
in 2.6.26 internal kernel's sysfs interface had a major change, which
made it heavily incompatible with pre-2.6.26 version. But with the
obsolete procfs interface kernels 2.6.16+ are supported.

If during compilation you see message like "*** No rule to make target
`xxx.h', needed by `yyy.o'.  Stop.", then your autogenerated
dependencies don't match your compiler configuration anymore. You should
run "make extraclean" to remove them. On the next compilation they will
be regenerated.

If you experience problems during kernel module load or running, check
your system and/or kernel logs (or run dmesg command for the few most
recent kernel messages).

To use full power of TCP zero-copy transmit functions, especially
dealing with user space supplied via scst_user module memory, iSCSI-SCST
needs to be notified when Linux networking finished data transmission.
For that you should enable CONFIG_TCP_ZERO_COPY_TRANSFER_COMPLETION_NOTIFICATION
kernel config option. This is highly recommended, but not required.
Basically, iSCSI-SCST works fine with an unpatched Linux kernel with the
same or better speed as other open source iSCSI targets, including IET,
but if you want even better performance you have to patch and rebuild
the kernel. Without CONFIG_TCP_ZERO_COPY_TRANSFER_COMPLETION_NOTIFICATION
enabled you will just revert to the original behavior of other open
source iSCSI targets, when for data transmission:

 - For in-kernel allocated memory (scst_vdisk and pass-through
   handlers) usage of SGV cache on transmit path (READ-type commands)
   will be disabled, but data will still be sent in zero-copy manner.

 - For user space allocated memory (scst_user handler) all transmitted
   data will be additionally copied into temporary TCP buffers. The
   performance hit will be quite noticeable.

Note, that if your network hardware does not support TX offload
functions or has them disabled, then TCP zero-copy transmit functions on
your system will not be used by Linux networking in any case, so
put_page_callback patch will not be able to improve performance for you.
You can check your network hardware offload capabilities by command
"ethtool -k ethX", where X is the network device number. At least
"tx-checksumming" and "scatter-gather" should be enabled.

If you have in your kernel log error messages like:

iscsi-scst: ***ERROR*** net_priv isn't NULL and != ref_cmd

with the corresponding kernel BUG dump, then put_page_callback patch you
use isn't sufficient for your kernel. This might be because the kernel
you use has some additional patches applied, which affect the
functionality, which put_page_callback patch provides. For example,
Fedora or Gentoo use kernels, which, although have version number like
2.6.18, are greatly differ from the "vanilla" kernel 2.6.18,
maintained by Linus Torvalds for that the put_page_callback patch was
created. In this case it is recommended either:

 - Search net/ in your kernel source for "put_page" and "get_page" functions.
   If you find any in some place, except in net/sunrpc/svc.c and
   net/core/pktgen.c, then, most likely, you found the reason of your
   problem. Replace them by "net_put_page" and "net_get_page"
   correspondingly and try again. If the problem is solved, then please
   prepare a new put_page_callback patch and send it to the SCST mailing
   list [email protected].

or

 - Unapply this patch and use iSCSI-SCST without it. Also report this
   problem to the SCST mailing list [email protected].


IMPORTANT: This patch does not support compound pages, so not working with
=========  network drivers using them. Sings of that is either when
	   iSCSI-SCST keeps closing connections, because no reply from
	   initiator for too long time, or system crashes on NULL
	   pointer dereference in iscsi_get_page_callback. You can find
	   list of such drivers by searching in drivers/net for
	   __GFP_COMP word in their source code.

Usage
-----

See HOWTOs in the doc/ subdirectory.

If you want to use Intel CRC32 offload and have corresponding hardware,
you should load crc32c-intel module. Then iSCSI-SCST will do all digest
calculations using this facility.

In 2.0.0 usage of iscsi-scstd.conf as well as iscsi-scst-adm utility is
obsolete. Use the sysfs interface facilities instead.

The flow of iSCSI-SCST inialization should be as the following:

1. Load of SCST and iSCSI-SCST kernel modules with necessary module
parameters, if needed.

2. Start iSCSI-SCST service.

3. Configure targets, devices, LUNs, etc. either using scstadmin
(recommended), or using the sysfs interface directly as described below.

It is recommended to use TEST UNIT READY ("tur") command to check if
iSCSI-SCST target is alive in MPIO configurations.

Also see SCST README file how to tune for the best performance.

CAUTION: Working of target and initiator on the same host isn't fully
=======  supported. See SCST README file for details.


Migration from the obsolete proc interface
------------------------------------------

Sysfs enabled scstadmin supports old procfs config file format, so with
it you should do the following steps to migrate your proc-based
configuration to the sysfs interface:

1. Load SCST modules

2. Run "scstadmin -config old_config_file"

3. Start iSCSI-SCST

4. Run "scstadmin -write_config new_config_file"

5. Check new_config_file and make sure it has everything written
properly.

6. IMPORTANT! Delete /etc/iscsi-scst.conf and forget about it.

7. Start using "scstadmin -config new_config_file" to configure both
SCST and iSCSI-SCST.

IMPORTANT: With the sysfs interface the flow of iSCSI-SCST initialization
=========  has changed (see above). Now scstadmin should be run as the FINAL
	   step of the initialization and only ONCE.

Also it is recommended to convert initiators.allow and initiators.deny
files to the corresponding sysfs facilities.


Migration from IET
------------------

Scalable Informatics Inc. has developed a pretty simple tool, will
automatically generate an /etc/scst.conf file from an /etc/ietd.conf
file, and then restart daemons.

Tool is here: http://download.scalableinformatics.com/gen_scst_conf/

Just run it on an ietd.conf based iSCSI target:

$ chmod +x gen_scst_conf.pl
$ sudo ./gen_scst_conf.pl

and your IET machine should now largely be an iSCSI-SCST machine. With
the caveat of a few IET elements not being supported.

Current limitations are one LUN per target, and not all LUN options
might be mapped well between the systems. This may change if there is
demand. For any questions and inquirys feel free to contact Joe Landman
<[email protected]> (and please provide a few sample
ietd.conf files for us to play with).


Sysfs interface
---------------

Starting from 2.0.0 iSCSI-SCST uses sysfs interface. The procfs
interface is obsolete and will be removed in one of the next versions.
The sysfs build supports only kernels 2.6.26 and higher, because in
2.6.26 internal kernel's sysfs interface had a major change, which made
it heavily incompatible with pre-2.6.26 version. But with the obsolete
procfs interface kernels 2.6.16+ are supported.

Root of SCST sysfs interface is /sys/kernel/scst_tgt. Root of iSCSI-SCST
is /sys/kernel/scst_tgt/targets/iscsi. It has the following entries:

 - None, one or more subdirectories for targets with name equal to names
   of the corresponding targets.

 - IncomingUser[num] - optional one or more attributes containing user
   name and password for incoming discovery user name. Not exist by
   default and can be added through "mgmt" entry, see below.

 - OutgoingUser - optional attribute containing user name and password
   for outgoing discovery user name. Not exist by default and can be
   added through "mgmt" entry, see below.

 - iSNSServer - contains name or IP address of iSNS server with optional
   "AccessControl" attribute, which allows to enable iSNS access
   control. Empty by default.

 - allowed_portal[num] - optional attribute, which specifies, on which
   portals (target's IP addresses) this target will be available. If not
   specified (default) the target will be available on all all portals.
   As soon as at least one allowed_portal specified, the target will be
   accessible for initiators only on the specified portals. There might
   be any number of the allowed_portal attributes. The portals
   specification in the allowed_portal attributes can be a simple
   DOS-type patterns, containing '*' and '?' symbols. '*' means match
   all any symbols, '?' means match only any single symbol. For
   instance, "10.170.77.2" will match "10.170.7?.*". Additionally, you
   can use negative sign '!' to revert the value of the pattern. For
   instance, "10.170.67.2" will match "!10.170.7?.*". See examples
   below.

 - enabled - using this attribute you can enable or disable iSCSI-SCST
   accept new connections. It allows to finish configuring global
   iSCSI-SCST attributes before it starts accepting new connections. 0
   by default.

 - open_state - read-only attribute, which allows to see if the user
   space part of iSCSI-SCST connected to the kernel part.

 - per_portal_acl - if set, makes iSCSI-SCST work in the per-portal
   access control mode. In this mode iSCSI-SCST registers all initiators
   in SCST core as "initiator_name#portal_IP_address" pattern, like
   "iqn.2006-10.net.vlnb:ini#10.170.77.2" for initiator
   iqn.2006-10.net.vlnb connected through portal 10.170.77.2. This mode
   allows to make particular initiators be able to use only particular
   portals on the target and don't see/be able to connect through
   others. See below for more details.

 - trace_level - allows to enable and disable various tracing
   facilities. See content of this file for help how to use it.

 - version - read-only attribute, which allows to see version of
   iSCSI-SCST and enabled optional features.

 - mgmt - main management entry, which allows to configure iSCSI-SCST.
   Namely, add/delete targets as well as add/delete optional global and
   per-target attributes. See content of this file for help how to use
   it.

Each iSCSI-SCST sysfs file (attribute) can contain in the last line mark
"[key]". It is automatically added mark used to allow scstadmin to see
which attributes it should save in the config file. You can ignore it.

Each target subdirectory contains the following entries:

 - ini_groups - subdirectory defining initiator groups for this target,
   used to define per-initiator access control. See SCST core README for
   more details.

 - luns - subdirectory defining LUNs of this target. See SCST core
   README for more details.

 - sessions - subdirectory containing connected to this target sessions.

 - IncomingUser[num] - optional one or more attributes containing user
   name and password for incoming user name. Not exist by default and can
   be added through the "mgmt" entry, see above.

 - OutgoingUser - optional attribute containing user name and password
   for outgoing user name. Not exist by default and can be added through
   the "mgmt" entry, see above.

 - Entries defining default iSCSI parameters values used during iSCSI
   parameters negotiation. Only entries which can be changed or make
   sense are listed there.

 - QueuedCommands - defines maximum number of commands queued to any
   session of this target. Default is 32 commands.

 - NopInInterval - defines interval between NOP-In requests, which the
   target will send on idle connections to check if the initiator is
   still alive. If there is no NOP-Out reply from the initiator in
   NopInTimeout seconds, the corresponding connection will be closed. Default
   is 30 seconds. If it's set to 0, then NOP-In requests are disabled.

 - NopInTimeout - defines the maximum time in seconds a NOP-In request
   can wait for response from initiator, otherwise the corresponding
   connection will be closed. Default is 30 seconds.

 - RspTimeout - defines the maximum time in seconds a command can wait for
   response from initiator, otherwise the corresponding connection will
   be closed. Default is 90 seconds.

 - enabled - using this attribute you can enable or disable iSCSI-SCST
   accept new connections to this target. It allows to finish
   configuring it before it starts accepting new connections. 0 by
   default.

 - redirect - allows to temporarily or permanently redirect login to the
   target to another portal. Discovery sessions will not be impacted,
   but normal sessions will be redirected before security negotiation.
   The destination should be specified using format "<ip_addr>[:port] temp|perm".
   IPv6 addresses need to be enclosed in [] brackets. To remove
   redirection, provide an empty string. For example:
   echo "10.170.77.2:32600 temp" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/redirect
   will temporarily redirect login to portal 10.170.77.2 and port 32600.

 - tid - TID of this target.

The "sessions" subdirectory contains the following attribute:

 - thread_pid - the process identifiers (PIDs) of the iscsird and iscsiwr
   threads that process SCSI commands associated with this session.

Additionally, the "sessions" subdirectory contains one subdirectory for each
connected session with name equal to name of the connected initiator.

Each session subdirectory contains the following entries:

 - One subdirectory for each TCP connection in this session. ISCSI-SCST
   supports 1 connection per session, but the session subdirectory can
   contain several connections: one active and other being closed.

 - Entries defining negotiated iSCSI parameters. Only parameters which
   can be changed or make sense are listed there.

 - initiator_name - contains initiator name

 - sid - contains SID of this session

 - reinstating - contains reinstatement state of this session

 - force_close - write-only attribute, which allows to force close this
   session. This is the only writable session attribute.

 - active_commands - contains number of active, i.e. not yet or being
   executed, SCSI commands in this session.

 - commands - contains overall number of SCSI commands in this session.

 - thread_pid - Process IDs (PIDs) of the iscsi{wr,rd} kernel threads that
   process the SCSI commands for this session.

Each connection subdirectory contains the following entries:

 - cid - contains CID of this connection.

 - ip - contains IP address of the connected initiator.

 - state - contains processing state of this connection.

Each initiator group subdirectory contains:

 - per_sess_dedicated_tgt_threads - if set, each iSCSI session has
   dedicated, i.e. not shared with other sessions, pool of the
   iscsi{wr,rd} kernel threads. Useful to control per-session CPU
   affinity to improve performance. Default: not set.

See SCST README for info about other attributes.

Below is a sample script, which configures 1 virtual disk "disk1" using
/disk1 image and one target iqn.2006-10.net.vlnb:tgt with all default
parameters:

#!/bin/bash

modprobe scst
modprobe scst_vdisk

echo "add_device disk1 filename=/disk1; nv_cache=1" >/sys/kernel/scst_tgt/handlers/vdisk_fileio/mgmt

service iscsi-scst start

echo "add_target iqn.2006-10.net.vlnb:tgt" >/sys/kernel/scst_tgt/targets/iscsi/mgmt
echo "add disk1 0" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/luns/mgmt

echo 1 >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/enabled
echo 1 >/sys/kernel/scst_tgt/targets/iscsi/enabled

Below is another sample script, which configures 1 real local SCSI disk
0:0:1:0 and one target iqn.2006-10.net.vlnb:tgt with all default parameters:

#!/bin/bash

modprobe scst
modprobe scst_disk

echo "add_device 0:0:1:0" >/sys/kernel/scst_tgt/handlers/dev_disk/mgmt

service iscsi-scst start

echo "add_target iqn.2006-10.net.vlnb:tgt" >/sys/kernel/scst_tgt/targets/iscsi/mgmt
echo "add 0:0:1:0 0" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/luns/mgmt

echo 1 >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/enabled
echo 1 >/sys/kernel/scst_tgt/targets/iscsi/enabled

Below is an advanced sample script, which configures more virtual
devices of various types, including virtual CDROM and 2 targets, one
with all default parameters, another one with some not default
parameters, incoming and outgoing user names for CHAP authentication,
and special permissions for initiator iqn.2005-03.org.open-iscsi:cacdcd2520,
which will see another set of devices. Also this sample configures CHAP
authentication for discovery sessions and iSNS server with access
control.

#!/bin/bash

modprobe scst
modprobe scst_vdisk

echo "add_device disk1 filename=/disk1; nv_cache=1" >/sys/kernel/scst_tgt/handlers/vdisk_fileio/mgmt
echo "add_device disk2 filename=/disk2; blocksize=4096; nv_cache=1" >/sys/kernel/scst_tgt/handlers/vdisk_fileio/mgmt
echo "add_device blockio filename=/dev/sda5" >/sys/kernel/scst_tgt/handlers/vdisk_blockio/mgmt
echo "add_device nullio" >/sys/kernel/scst_tgt/handlers/vdisk_nullio/mgmt
echo "add_device cdrom" >/sys/kernel/scst_tgt/handlers/vcdrom/mgmt

service iscsi-scst start

echo "192.168.1.16 AccessControl" >/sys/kernel/scst_tgt/targets/iscsi/iSNSServer
echo "add_attribute IncomingUser joeD 12charsecret" >/sys/kernel/scst_tgt/targets/iscsi/mgmt
echo "add_attribute OutgoingUser jackD 12charsecret1" >/sys/kernel/scst_tgt/targets/iscsi/mgmt

echo "add_target iqn.2006-10.net.vlnb:tgt" >/sys/kernel/scst_tgt/targets/iscsi/mgmt

echo "add disk1 0" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/luns/mgmt
echo "add cdrom 1" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/luns/mgmt

echo "add_target iqn.2006-10.net.vlnb:tgt1" >/sys/kernel/scst_tgt/targets/iscsi/mgmt
echo "add_target_attribute iqn.2006-10.net.vlnb:tgt1 IncomingUser1 joe2 12charsecret2" >/sys/kernel/scst_tgt/targets/iscsi/mgmt
echo "add_target_attribute iqn.2006-10.net.vlnb:tgt1 IncomingUser joe 12charsecret" >/sys/kernel/scst_tgt/targets/iscsi/mgmt
echo "add_target_attribute iqn.2006-10.net.vlnb:tgt1 OutgoingUser jim1 12charpasswd" >/sys/kernel/scst_tgt/targets/iscsi/mgmt
echo "No" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/InitialR2T
echo "Yes" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/ImmediateData
echo "8192" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/MaxRecvDataSegmentLength
echo "8192" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/MaxXmitDataSegmentLength
echo "131072" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/MaxBurstLength
echo "32768" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/FirstBurstLength
echo "1" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/MaxOutstandingR2T
echo "CRC32C,None" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/HeaderDigest
echo "CRC32C,None" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/DataDigest
echo "32" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/QueuedCommands

echo "add disk2 0" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/luns/mgmt
echo "add nullio 26" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/luns/mgmt

echo "create special_ini" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/ini_groups/mgmt
echo "add blockio 0 read_only=1" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/ini_groups/special_ini/luns/mgmt
echo "add iqn.2005-03.org.open-iscsi:cacdcd2520" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/ini_groups/special_ini/initiators/mgmt

echo 1 >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/enabled
echo 1 >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/enabled

echo 1 >/sys/kernel/scst_tgt/targets/iscsi/enabled

The resulting overall SCST sysfs hierarchy with an initiator connected to
both iSCSI-SCST targets will look like:

/sys/kernel/scst_tgt
|-- devices
|   |-- blockio
|   |   |-- blocksize
|   |   |-- exported
|   |   |   `-- export0 -> ../../../targets/iscsi/iqn.2006-10.net.vlnb:tgt1/ini_groups/special_ini/luns/0
|   |   |-- filename
|   |   |-- handler -> ../../handlers/vdisk_blockio
|   |   |-- nv_cache
|   |   |-- read_only
|   |   |-- removable
|   |   |-- resync_size
|   |   |-- size_mb
|   |   |-- t10_dev_id
|   |   |-- threads_num
|   |   |-- threads_pool_type
|   |   |-- type
|   |   `-- usn
|   |-- cdrom
|   |   |-- exported
|   |   |   `-- export0 -> ../../../targets/iscsi/iqn.2006-10.net.vlnb:tgt/luns/1
|   |   |-- filename
|   |   |-- handler -> ../../handlers/vcdrom
|   |   |-- size_mb
|   |   |-- t10_dev_id
|   |   |-- threads_num
|   |   |-- threads_pool_type
|   |   |-- type
|   |   `-- usn
|   |-- disk1
|   |   |-- blocksize
|   |   |-- exported
|   |   |   `-- export0 -> ../../../targets/iscsi/iqn.2006-10.net.vlnb:tgt/luns/0
|   |   |-- filename
|   |   |-- handler -> ../../handlers/vdisk_fileio
|   |   |-- nv_cache
|   |   |-- o_direct
|   |   |-- read_only
|   |   |-- removable
|   |   |-- resync_size
|   |   |-- size_mb
|   |   |-- t10_dev_id
|   |   |-- threads_num
|   |   |-- threads_pool_type
|   |   |-- type
|   |   |-- usn
|   |   `-- write_through
|   |-- disk2
|   |   |-- blocksize
|   |   |-- exported
|   |   |   `-- export0 -> ../../../targets/iscsi/iqn.2006-10.net.vlnb:tgt1/luns/0
|   |   |-- filename
|   |   |-- handler -> ../../handlers/vdisk_fileio
|   |   |-- nv_cache
|   |   |-- o_direct
|   |   |-- read_only
|   |   |-- removable
|   |   |-- resync_size
|   |   |-- size_mb
|   |   |-- t10_dev_id
|   |   |-- threads_num
|   |   |-- threads_pool_type
|   |   |-- type
|   |   |-- usn
|   |   `-- write_through
|   `-- nullio
|       |-- blocksize
|       |-- exported
|       |   `-- export0 -> ../../../targets/iscsi/iqn.2006-10.net.vlnb:tgt1/luns/26
|       |-- handler -> ../../handlers/vdisk_nullio
|       |-- read_only
|       |-- removable
|       |-- size_mb
|       |-- t10_dev_id
|       |-- threads_num
|       |-- threads_pool_type
|       |-- type
|       `-- usn
|-- handlers
|   |-- vcdrom
|   |   |-- cdrom -> ../../devices/cdrom
|   |   |-- mgmt
|   |   |-- trace_level
|   |   `-- type
|   |-- vdisk_blockio
|   |   |-- blockio -> ../../devices/blockio
|   |   |-- mgmt
|   |   |-- trace_level
|   |   `-- type
|   |-- vdisk_fileio
|   |   |-- disk1 -> ../../devices/disk1
|   |   |-- disk2 -> ../../devices/disk2
|   |   |-- mgmt
|   |   |-- trace_level
|   |   `-- type
|   `-- vdisk_nullio
|       |-- mgmt
|       |-- nullio -> ../../devices/nullio
|       |-- trace_level
|       `-- type
|-- sgv
|   |-- global_stats
|   |-- sgv
|   |   `-- stats
|   |-- sgv-clust
|   |   `-- stats
|   `-- sgv-dma
|       `-- stats
|-- targets
|   `-- iscsi
|       |-- IncomingUser
|       |-- OutgoingUser
|       |-- enabled
|       |-- iSNSServer
|       |-- iqn.2006-10.net.vlnb:tgt
|       |   |-- DataDigest
|       |   |-- FirstBurstLength
|       |   |-- HeaderDigest
|       |   |-- ImmediateData
|       |   |-- InitialR2T
|       |   |-- MaxBurstLength
|       |   |-- MaxOutstandingR2T
|       |   |-- MaxRecvDataSegmentLength
|       |   |-- MaxXmitDataSegmentLength
|       |   |-- NopInInterval
|       |   |-- QueuedCommands
|       |   |-- RspTimeout
|       |   |-- enabled
|       |   |-- ini_groups
|       |   |   `-- mgmt
|       |   |-- luns
|       |   |   |-- 0
|       |   |   |   |-- device -> ../../../../../devices/disk1
|       |   |   |   `-- read_only
|       |   |   |-- 1
|       |   |   |   |-- device -> ../../../../../devices/cdrom
|       |   |   |   `-- read_only
|       |   |   `-- mgmt
|       |   |-- per_portal_acl
|       |   |-- redirect
|       |   |-- rel_tgt_id
|       |   |-- sessions
|       |   |   `-- iqn.2005-03.org.open-iscsi:cacdcd2520
|       |   |       |-- 10.170.75.2
|       |   |       |   |-- cid
|       |   |       |   |-- ip
|       |   |       |   `-- state
|       |   |       |-- DataDigest
|       |   |       |-- FirstBurstLength
|       |   |       |-- HeaderDigest
|       |   |       |-- ImmediateData
|       |   |       |-- InitialR2T
|       |   |       |-- MaxBurstLength
|       |   |       |-- MaxOutstandingR2T
|       |   |       |-- MaxRecvDataSegmentLength
|       |   |       |-- MaxXmitDataSegmentLength
|       |   |       |-- active_commands
|       |   |       |-- commands
|       |   |       |-- force_close
|       |   |       |-- initiator_name
|       |   |       |-- luns -> ../../luns
|       |   |       |-- reinstating
|       |   |       `-- sid
|       |   `-- tid
|       |-- iqn.2006-10.net.vlnb:tgt1
|       |   |-- DataDigest
|       |   |-- FirstBurstLength
|       |   |-- HeaderDigest
|       |   |-- ImmediateData
|       |   |-- IncomingUser
|       |   |-- IncomingUser1
|       |   |-- InitialR2T
|       |   |-- MaxBurstLength
|       |   |-- MaxOutstandingR2T
|       |   |-- MaxRecvDataSegmentLength
|       |   |-- MaxXmitDataSegmentLength
|       |   |-- OutgoingUser
|       |   |-- NopInInterval
|       |   |-- QueuedCommands
|       |   |-- RspTimeout
|       |   |-- enabled
|       |   |-- ini_groups
|       |   |   |-- mgmt
|       |   |   `-- special_ini
|       |   |       |-- initiators
|       |   |       |   |-- iqn.2005-03.org.open-iscsi:cacdcd2520
|       |   |       |   `-- mgmt
|       |   |       `-- luns
|       |   |           |-- 0
|       |   |           |   |-- device -> ../../../../../../../devices/blockio
|       |   |           |   `-- read_only
|       |   |           `-- mgmt
|       |   |-- luns
|       |   |   |-- 0
|       |   |   |   |-- device -> ../../../../../devices/disk2
|       |   |   |   `-- read_only
|       |   |   |-- 26
|       |   |   |   |-- device -> ../../../../../devices/nullio
|       |   |   |   `-- read_only
|       |   |   `-- mgmt
|       |   |-- per_portal_acl
|       |   |-- redirect
|       |   |-- rel_tgt_id
|       |   |-- sessions
|       |   |   `-- iqn.2005-03.org.open-iscsi:cacdcd2520
|       |   |       |-- 10.170.75.2
|       |   |       |   |-- cid
|       |   |       |   |-- ip
|       |   |       |   `-- state
|       |   |       |-- DataDigest
|       |   |       |-- FirstBurstLength
|       |   |       |-- HeaderDigest
|       |   |       |-- ImmediateData
|       |   |       |-- InitialR2T
|       |   |       |-- MaxBurstLength
|       |   |       |-- MaxOutstandingR2T
|       |   |       |-- MaxRecvDataSegmentLength
|       |   |       |-- MaxXmitDataSegmentLength
|       |   |       |-- active_commands
|       |   |       |-- commands
|       |   |       |-- force_close
|       |   |       |-- initiator_name
|       |   |       |-- luns -> ../../ini_groups/special_ini/luns
|       |   |       |-- reinstating
|       |   |       `-- sid
|       |   `-- tid
|       |-- mgmt
|       |-- open_state
|       |-- trace_level
|       `-- version
|-- threads
|-- trace_level
`-- version


Configuring iscsi-scst via scstadmin
------------------------------------

All iscsi-scst parameters that were configured in the previous section via the
sysfs interface can also be set via scstadmin. Here is an example that shows
how to configure the IP address of the iSNSServer and to enable iSNS access
control at the same time:

# scstadmin -noprompt -set_drv_attr iscsi -attributes iSNSServer="192.168.1.16 AccessControl"

Collecting current configuration: done.


-> Making requested changes.
        -> Done, 0 change(s) made.

All done.

# scstadmin -list_drv_attr iscsi

Collecting current configuration: done.

        Attribute      Value                       Writable      KEY
        ------------------------------------------------------------
        iSNSServer     127.0.0.1 AccessControl     Yes           Yes

        Dynamic attributes available
        ----------------------------
        OutgoingUser
        IncomingUser

All done.


Advanced initiators access control
----------------------------------

ISCSI-SCST allows you to optionally control visibility and accessibility
of your target and its portals (IP addresses) to remote initiators. This
control includes both the target's portals SendTargets discovery as well
as regular LUNs access.

This facility supersedes the obsolete initiators.[allow,deny] method,
which is going to be removed in one of the future versions.

This facility is available only in the sysfs build of iSCSI-SCST.

By default, all portals are available for the initiators.

1. If you want to enable/disable one or more target's portals for all
initiators, you should define one ore more allowed_portal attributes.
For example:

echo 'add_target_attribute iqn.2006-10.net.vlnb:tgt allowed_portal 10.170.77.2' >/sys/kernel/scst_tgt/targets/iscsi/mgmt

will enable only portal 10.170.77.2 and disable all other portals

echo 'add_target_attribute iqn.2006-10.net.vlnb:tgt allowed_portal 10.170.77.2' >/sys/kernel/scst_tgt/targets/iscsi/mgmt
echo 'add_target_attribute iqn.2006-10.net.vlnb:tgt allowed_portal 10.170.75.2' >/sys/kernel/scst_tgt/targets/iscsi/mgmt

will enable only portals 10.170.77.2 and 10.170.75.2 and disable all
other portals.

echo 'add_target_attribute iqn.2006-10.net.vlnb:tgt allowed_portal 10.170.7?.2' >/sys/kernel/scst_tgt/targets/iscsi/mgmt

will enable only portals 10.170.7x.2 and disable all other portals.

echo 'add_target_attribute iqn.2006-10.net.vlnb:tgt allowed_portal !*' >/sys/kernel/scst_tgt/targets/iscsi/mgmt

will disable all portals.

2. If you want to want to allow only only specific set of initiators be
able to connect to your target, you should don't add any default LUNs
for the target and create for allowed initiators a security group to
which they will be assigned.

For example, we want initiator iqn.2005-03.org.vlnb:cacdcd2520 and only
it be able to access target iqn.2006-10.net.vlnb:tgt:

echo 'add_target iqn.2006-10.net.vlnb:tgt' >/sys/kernel/scst_tgt/targets/iscsi/mgmt
echo 'create allowed_ini' >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/ini_groups/mgmt
echo 'add dev1 0' >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/ini_groups/allowed_ini/luns/mgmt
echo 'add iqn.2005-03.org.vlnb:cacdcd2520' >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/ini_groups/allowed_ini/initiators/mgmt
echo 1 >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/enabled

Since there will be no default LUNs for the target, all initiators other
than iqn.2005-03.org.vlnb:cacdcd2520 will be blocked from accessing it.

Alternatively, you can create an empty security group and filter out in
it all initiators except the allowed one:

echo 'add_target iqn.2006-10.net.vlnb:tgt' >/sys/kernel/scst_tgt/targets/iscsi/mgmt
echo 'add dev1 0' >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/luns/mgmt
echo 'create denied_inis' >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/ini_groups/mgmt
echo 'add !iqn.2005-03.org.vlnb:cacdcd2520' >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/ini_groups/denied_inis/initiators/mgmt
echo 1 >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/enabled

3. If you want to enable/disable one or more target's portals for
particular initiators, you should set per_portal_acl attribute to 1 and
specify SCST access control to those initiators. If an SCST security
group doesn't have any LUNs, all the initiator, which should be assigned
to it, will not see this target and/or its portal. For example:

(We assume that an empty group "BLOCKING_GROUP" is already created by for
target iqn.2006-10.net.vlnb:tgt by command (see above for more information):
"echo 'create BLOCKING_GROUP' >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/ini_groups/mgmt)

echo 'add iqn.2005-03.org.vlnb:cacdcd2520#10.170.77.2' >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/ini_groups/BLOCKING_GROUP/initiators/mgmt

will block access of initiator iqn.2005-03.org.vlnb:cacdcd2520 to
target iqn.2006-10.net.vlnb:tgt portal 10.170.77.2.

Another example:

echo 'add iqn.2005-03.org.vlnb:cacdcd2520*' >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/ini_groups/BLOCKING_GROUP/initiators/mgmt

will block access of initiator iqn.2005-03.org.vlnb:cacdcd2520 to
all target iqn.2006-10.net.vlnb:tgt portals.


Troubleshooting
---------------

If you have any problems, start troubleshooting from looking at the
kernel and system logs. In the kernel log iSCSI-SCST and SCST core send
their messages, in the system log iscsi-scstd sends its messages. In
most Linux distributions both those logs are put to /var/log/messages
file.

Then, it might be helpful to increase level of logging. For kernel
modules you should make the debug build, by either running "make 2debug"
if you work with SCST SVN tree, or by enabling the corresponding
debug symbols (see below).

If after looking on the logs the reason of your problem is still unclear
for you, report to SCST mailing list [email protected].


Work if target's backstorage or link is too slow
------------------------------------------------

In some cases you can experience I/O stalls or see in the kernel log
abort or reset messages. It can happen under high I/O load, when your
target's backstorage gets overloaded, or working over a slow link, when
the link can't serve all the queued commands on time,

To workaround it you can reduce QueuedCommands parameter for the
corresponding target to some lower value, like 8 (default is 32).

Also see SCST README file for more details about that issue and ways to
prevent it.


Performance advices
-------------------

1. If you use Windows XP or Windows 2003+ as initiators, you can
consider to decrease TcpAckFrequency parameter to 1. See
http://support.microsoft.com/kb/328890/ or google for "TcpAckFrequency"
for more details.

2. See how to get the maximum throughput from iSCSI, for instance, at
http://virtualgeek.typepad.com/virtual_geek/2009/01/a-multivendor-post-to-help-our-mutual-iscsi-customers-using-vmware.html.
It's about VMware, but its recommendations apply to other environments
as well.

3. ISCSI initiators from pre-CentOS/RHEL 5 reported to have some
performance problems. If you use it, it is strongly advised to upgrade.

4. If you are going to use your target in an VM environment, for
instance as a shared storage with VMware, make sure all your VMs
connected to the target via *separate* sessions, i.e. each VM has own
connection to the target, not all VMs connected using a single
connection. You can check it using SCST proc or sysfs interface. If you
miss it, you can greatly loose performance of parallel access to your
target from different VMs. This isn't related to the case if your VMs
are using the same shared storage, like with VMFS, for instance. In this
case all your VM hosts will be connected to the target via separate
sessions, which is enough.

5. Many dual port network adapters are not able to transfer data
simultaneously on both ports, i.e. they transfer data via both ports on
the same speed as via any single port. Thus, using such adapters in MPIO
configuration can't improve performance. To allow MPIO to have double
performance you should either use separate network adapters, or find a
dual-port adapter capable to to transfer data simultaneously on both
ports. You can check it by running 2 iperf's through both ports in
parallel.

6. Since network offload works much better in the write direction, than
for reading (simplifying, in the read direction often there's additional
data copy) in many cases with 10GbE in a single initiator-target pair
the initiator's CPU is a bottleneck, so you can see the initiator can
read data on much slower rate, than write. You can check it by watching
*each particular* CPU load to find out if any of them is close to 100%
load, including IRQ processing load. Note, many tools like vmstat give
aggregate load on all CPUs, so with 4 cores 25% corresponds to 100% load
of any single CPU.

7. For high speed network adapters it can be better if you configure
them to serve connections, e.g., from initiator on CPU0 and from
initiator Y on CPU1. Then you can bind threads processing them also to
CPU0 and CPU1 correspondingly using cpu_mask attribute of their targets
or security groups. In NUMA-like configurations it can signficantly
boost IOPS performance.

8. See SCST core's README for more advices. Especially pay attention to
have io_grouping_type option set correctly.


Compilation options
-------------------

There are the following compilation options, that could be commented
in/out in the kernel's module Makefile:

 - CONFIG_SCST_DEBUG - turns on some debugging code, including some logging.
   Makes the driver considerably bigger and slower, producing large amount of
   log data.

 - CONFIG_SCST_TRACING - turns on ability to log events. Makes the driver
   considerably bigger and leads to some performance loss.

 - CONFIG_SCST_EXTRACHECKS - adds extra validity checks in the various places.

 - CONFIG_SCST_ISCSI_DEBUG_DIGEST_FAILURES - simulates digest failures in
   random places.


Creating version of put_page_callback patch for your kernel
-----------------------------------------------------------

If you need your own version of put_page_callback patch for your custom
kernel, for which there is no prepared version, you can create it
yourself. This is pretty mechanical work, you don't need to understand
how it works, you only need to do the following two steps:

1. Apply the closest version of put_page_callback-<kernel-version>.patch
on your kernel. Resolve only failed hunks from include/ and
net/core/utils.c, ignore other failures.

2. Search net/ in your kernel source for "put_page" and "get_page"
functions. Replace them by "net_put_page" and "net_get_page"
correspondingly.

For vanilla kernels that should be all. Then please send your new
put_page_callback-<kernel-version>.patch to the SCST mailing list
[email protected].

But some out of tree drivers (some versions of DRBD and Intel e1000 are
reported to do so) use own pages referencing on the TX path, so they
should be modified the same way as above as well. (No need to modify RX
path, because put_page_callback logic used by SCST only on the TX path.)

In case of DRBD, if you don't do this modification, you can specify
disable_sendpage parameter for drbd module. But in this case it might be
better for performance simply not apply put_page_callback patch.


Background information about zero-copy data sending
---------------------------------------------------

As explained above the most efficient operation of the iSCSI-SCST target
driver is achieved when the following two conditions are met:
* Data is sent from target to initiator in a zero-copy fashion.
* Data buffers are cached for reuse (by the so-called sgv pool). 
Unfortunately the zero-copy API in the Linux kernel (proto.sendpage() /
tcp_sendpage()) does not yet support completion notifications. Hence the
put_page_callback patch which adds completion notification support to
tcp_sendpage(). However, since the put_page_callback patch increases the size
of the page structure in the Linux kernel that patch is considered
unacceptable for integration in the mainline kernel [1]. Another approach,
called skb paged fragment destructors, might get merged in the mainline kernel
in the future. Version five of that patch series has been posted in May 2012
[2].

Notes:
* The current implementation of scst_user is such that iSCSI-SCST only sends
  data allocated by scst_user in a zero-copy fashion if the put_page_callback
  patch has been applied. This holds even if scst_user did not use the sgv
  pool to allocate a data buffer.
* Zero-copy sending is only possible with network interface drivers that
  support scatter/gather and checksumming (NETIF_F_SG and NETIF_F_ALL_CSUM
  respectively). 

References:
[1] James Bottomley, Linux Kernel Mailing List, December 2008,
    http://lkml.org/lkml/2008/12/11/213.
[2] Alex Bligh, Rebase Ian Campbell's skb fragment tracking to 3.2, January
    2013, linux-netdev, http://thread.gmane.org/gmane.linux.network/256820/.
[3] Ian Campbell, [PATCH v5 0/9] skb paged fragment destructors, May 3 2012,
    linux-netdev, http://thread.gmane.org/gmane.linux.network/229669.


Credits
-------

Thanks to:

 * Ming Zhang <[email protected]> for fixes

 * Krzysztof Blaszkowski <[email protected]> for many fixes

 * Alexey Kuznetsov <[email protected]> for comments and help in
   debugging

 * Tomasz Chmielewski <[email protected]> for testing and suggestions

 * Bart Van Assche <[email protected]> for a lot of help

Vladislav Bolkhovitin <[email protected]>, http://scst.sourceforge.net