7 patches for repository http://code.haskell.org/hackage-server: Tue Sep 27 09:59:51 BST 2011 Max Bolingbroke * Have the MirrorClient use the system proxy Tue Sep 27 10:35:04 BST 2011 Max Bolingbroke * Split out client functionality unrelated to mirroring from MirrorClient Tue Sep 27 10:37:56 BST 2011 Max Bolingbroke * Turn on some warnings for hackage-mirror and fix them Tue Sep 27 11:22:17 BST 2011 Max Bolingbroke * Rough template of Hackage build client Wed Sep 28 13:28:14 BST 2011 Max Bolingbroke * Bump acid-state and happstack versions to avoid bug There is a bug in happstack-server before v6.2.2 when you try to read the cookies/request params of a POST/PUT request with a content-type different from multipart/form-data or application/x-www-form-urlencoded The bug is described in http://comments.gmane.org/gmane.comp.lang.haskell.web/2033 As a result I changed the version bounds to v6.2.* Wed Sep 28 22:13:52 BST 2011 Max Bolingbroke * A mostly functional build client Wed Sep 28 22:17:54 BST 2011 Max Bolingbroke * Comments/debug code only New patches: [Have the MirrorClient use the system proxy Max Bolingbroke **20110927085951 Ignore-this: c2f1dc9d7ae36a9acc200a4f527f432 ] hunk ./MirrorClient.hs 293 setErrHandler die setOutHandler (debug verbosity) setAllowBasicAuth True + setCheckForProxy True action downloadFile :: URI -> FilePath -> HttpSession () [Split out client functionality unrelated to mirroring from MirrorClient Max Bolingbroke **20110927093504 Ignore-this: bfc13126ed4e974c460a916fab313c33 ] { addfile ./Distribution/Client.hs hunk ./Distribution/Client.hs 1 +{-# LANGUAGE PatternGuards #-} +module Distribution.Client where hunk ./Distribution/Client.hs 4 +import Network.HTTP +import Network.Browser +import Network.URI (URI(..), URIAuth(..), parseURI) + +import Distribution.Server.LegacyImport.UploadLog as UploadLog (read, Entry(..)) +import Distribution.Server.Users.Types (UserId(..), UserName(UserName)) +import Distribution.Server.Util.Index as PackageIndex (read) +import Distribution.Server.Util.Merge +import Distribution.Package +import Distribution.Verbosity +import Distribution.Simple.Utils + +import Data.List +import Data.Maybe +import Control.Applicative +import Control.Exception +import Data.Time +import Data.Time.Clock.POSIX +import Data.ByteString.Lazy.Char8 (ByteString) +import qualified Data.ByteString.Lazy.Char8 as BS +import qualified Codec.Compression.GZip as GZip +import qualified Codec.Archive.Tar as Tar +import qualified Codec.Archive.Tar.Entry as Tar + +import System.IO +import System.IO.Error +import System.FilePath +import System.Directory +import qualified System.FilePath.Posix as Posix + +------------------------- +-- Command line handling +------------------------- + +validateHackageURI :: String -> Either String URI +validateHackageURI str = case parseURI str of + Nothing -> Left ("invalid URL " ++ str) + Just uri + | uriScheme uri /= "http:" -> Left ("only http URLs are supported " ++ str) + | isNothing (uriAuthority uri) -> Left ("server name required in URL " ++ str) + | otherwise -> Right uri + +---------------------------------------------------- +-- Fetching info from source and destination servers +---------------------------------------------------- + +data PkgIndexInfo = PkgIndexInfo + PackageId + (Maybe UTCTime) + (Maybe UserName) + (Maybe UserId) + deriving Show + +downloadIndex :: URI -> FilePath -> HttpSession [PkgIndexInfo] +downloadIndex uri | isOldHackageURI uri = downloadOldIndex uri + | otherwise = downloadNewIndex uri + where + +isOldHackageURI :: URI -> Bool +isOldHackageURI uri + | Just auth <- uriAuthority uri = uriRegName auth == "hackage.haskell.org" + | otherwise = False + + +downloadOldIndex :: URI -> FilePath -> HttpSession [PkgIndexInfo] +downloadOldIndex uri cacheFile = do + + downloadFile indexURI indexFile + downloadFile logURI logFile + + ioAction $ do + + pkgids <- withFile indexFile ReadMode $ \hnd -> do + content <- BS.hGetContents hnd + case PackageIndex.read (\pkgid _ -> pkgid) (GZip.decompress content) of + Left err -> die $ "Error parsing index at " ++ show uri ++ ": " ++ err + Right pkgs -> return pkgs + + log <- withFile logFile ReadMode $ \hnd -> do + content <- hGetContents hnd + case UploadLog.read content of + Right log -> return log + Left err -> die $ "Error parsing log at " ++ show uri ++ ": " ++ err + + return (mergeLogInfo pkgids log) + + where + indexURI = uri "packages" "archive" "00-index.tar.gz" + indexFile = cacheFile <.> "tar.gz" + + logURI = uri "packages" "archive" "log" + logFile = cacheFile <.> "log" + + mergeLogInfo pkgids log = + catMaybes + . map selectDetails + $ mergeBy (\pkgid entry -> compare pkgid (entryPkgId entry)) + (sort pkgids) + ( map (maximumBy (comparing entryTime)) + . groupBy (equating entryPkgId) + . sortBy (comparing entryPkgId) + $ log ) + + selectDetails (OnlyInRight _) = Nothing + selectDetails (OnlyInLeft pkgid) = + Just $ PkgIndexInfo pkgid Nothing Nothing Nothing + selectDetails (InBoth pkgid (UploadLog.Entry time uname _)) = + Just $ PkgIndexInfo pkgid (Just time) (Just uname) Nothing + + entryPkgId (Entry _ _ pkgid) = pkgid + entryTime (Entry time _ _) = time + + +downloadNewIndex :: URI -> FilePath -> HttpSession [PkgIndexInfo] +downloadNewIndex uri cacheFile = do + downloadFile indexURI indexFile + ioAction $ withFile indexFile ReadMode $ \hnd -> do + content <- BS.hGetContents hnd + case PackageIndex.read selectDetails (GZip.decompress content) of + Left err -> error $ "Error parsing index at " ++ show uri ++ ": " ++ err + Right pkgs -> return pkgs + + where + indexURI = uri "packages/00-index.tar.gz" + indexFile = cacheFile <.> "tar.gz" + + selectDetails :: PackageId -> Tar.Entry -> PkgIndexInfo + selectDetails pkgid entry = + PkgIndexInfo + pkgid + (Just time) + (if null username then Nothing else Just (UserName username)) + (if userid == 0 then Nothing else Just (UserId userid)) + where + time = epochTimeToUTC (Tar.entryTime entry) + username = Tar.ownerName (Tar.entryOwnership entry) + userid = Tar.ownerId (Tar.entryOwnership entry) + + epochTimeToUTC :: Tar.EpochTime -> UTCTime + epochTimeToUTC = posixSecondsToUTCTime . realToFrac + + +------------------------- +-- HTTP utilities +------------------------- + +infixr 5 + +() :: URI -> FilePath -> URI +uri path = uri { uriPath = Posix.addTrailingPathSeparator (uriPath uri) + Posix. path } + + +type HttpSession a = BrowserAction (HandleStream ByteString) a + +httpSession :: Verbosity -> HttpSession a -> IO a +httpSession verbosity action = + browse $ do + setUserAgent "hackage-mirror" + setErrHandler die + setOutHandler (debug verbosity) + setAllowBasicAuth True + setCheckForProxy True + action + +downloadFile :: URI -> FilePath -> HttpSession () +downloadFile uri file = do + out $ "downloading " ++ show uri ++ " to " ++ file + let etagFile = file <.> "etag" + metag <- ioAction $ catchJustDoesNotExistError + (Just <$> readFile etagFile) + (\_ -> return Nothing) + case metag of + Just etag -> do + let headers = [mkHeader HdrIfNoneMatch (quote etag)] + (_, rsp) <- request (Request uri GET headers BS.empty) + case rspCode rsp of + (3,0,4) -> out $ file ++ " unchanged with ETag " ++ etag + (2,0,0) -> ioAction $ writeDowloadedFileAndEtag rsp + _ -> err (showFailure uri rsp) + + Nothing -> do + (_, rsp) <- request (Request uri GET [] BS.empty) + case rspCode rsp of + (2,0,0) -> ioAction $ writeDowloadedFileAndEtag rsp + _ -> err (showFailure uri rsp) + + where + writeDowloadedFileAndEtag rsp = do + BS.writeFile file (rspBody rsp) + setETag file (unquote <$> findHeader HdrETag rsp) + +getETag :: FilePath -> IO (Maybe String) +getETag file = + catchJustDoesNotExistError + (Just <$> readFile (file ".etag")) + (\_ -> return Nothing) + +setETag :: FilePath -> Maybe String -> IO () +setETag file Nothing = catchJustDoesNotExistError + (removeFile (file <.> "etag")) + (\_ -> return ()) +setETag file (Just etag) = writeFile (file <.> "etag") etag + +catchJustDoesNotExistError = + catchJust (\e -> if isDoesNotExistError e then Just e else Nothing) + +quote s = '"' : s ++ ['"'] + +unquote ('"':s) = go s + where + go [] = [] + go ('"':[]) = [] + go (c:cs) = c : go cs +unquote s = s + +-- AAARG! total lack of exception handling in HTTP monad! +downloadFile' :: URI -> FilePath -> HttpSession Bool +downloadFile' uri file = do + out $ "downloading " ++ show uri ++ " to " ++ file + mcontent <- requestGET' uri + case mcontent of + Nothing -> do out $ "404 " ++ show uri + return False + + Just content -> do ioAction $ BS.writeFile file content + return True + +requestGET :: URI -> HttpSession ByteString +requestGET uri = do + (_, rsp) <- request (Request uri GET headers BS.empty) + checkStatus uri rsp + return (rspBody rsp) + where + headers = [] + +-- Really annoying! +requestGET' :: URI -> HttpSession (Maybe ByteString) +requestGET' uri = do + (_, rsp) <- request (Request uri GET headers BS.empty) + case rspCode rsp of + (4,0,4) -> return Nothing + _ -> do checkStatus uri rsp + return (Just (rspBody rsp)) + where + headers = [] + + +requestPUT :: URI -> String -> ByteString -> HttpSession () +requestPUT uri mimetype body = do + (_, rsp) <- request (Request uri PUT headers body) + checkStatus uri rsp + where + headers = [ Header HdrContentLength (show (BS.length body)) + , Header HdrContentType mimetype ] + + +checkStatus :: URI -> Response ByteString -> HttpSession () +checkStatus uri rsp = case rspCode rsp of + (2,0,0) -> return () + (4,0,0) -> ioAction (warn normal (showFailure uri rsp)) >> return () + code -> err (showFailure uri rsp) + +showFailure uri rsp = + show (rspCode rsp) ++ " " ++ rspReason rsp ++ show uri + ++ case lookupHeader HdrContentType (rspHeaders rsp) of + Just mimetype | "text/plain" `isPrefixOf` mimetype + -> '\n' : BS.unpack (rspBody rsp) + _ -> "" hunk ./MirrorClient.hs 4 {-# LANGUAGE PatternGuards #-} module Main where -import Network.HTTP import Network.Browser hunk ./MirrorClient.hs 5 -import Network.URI (URI(..), URIAuth(..), parseURI, escapeURIString) +import Network.URI (URI(..), URIAuth(..)) hunk ./MirrorClient.hs 7 -import Distribution.Server.LegacyImport.UploadLog as UploadLog (read, Entry(..)) -import Distribution.Server.Users.Types (UserId(..), UserName(UserName)) -import Distribution.Server.Util.Index as PackageIndex (read) +import Distribution.Client import Distribution.Server.Util.Merge import Distribution.Package import Distribution.Version hunk ./MirrorClient.hs 18 import Data.List import Data.Maybe import Control.Monad -import Control.Applicative -import Control.Exception -import Data.Time -import Data.Time.Clock.POSIX -import Data.ByteString.Lazy.Char8 (ByteString) import qualified Data.ByteString.Lazy.Char8 as BS hunk ./MirrorClient.hs 19 -import qualified Codec.Compression.GZip as GZip -import qualified Codec.Archive.Tar as Tar -import qualified Codec.Archive.Tar.Entry as Tar import System.Environment hunk ./MirrorClient.hs 21 -import System.IO -import System.IO.Error import System.Exit import System.FilePath import System.Directory hunk ./MirrorClient.hs 25 import System.Console.GetOpt -import qualified System.FilePath.Posix as Posix data MirrorOpts = MirrorOpts { hunk ./MirrorClient.hs 34 selectedPkgs :: [PackageId] } -data PkgMirrorInfo = PkgMirrorInfo - PackageId - (Maybe UTCTime) - (Maybe UserName) - (Maybe UserId) - deriving Show - main :: IO () main = topHandler $ do args <- getArgs hunk ./MirrorClient.hs 74 mkCacheFileName uri = error $ "unexpected URI " ++ show uri -diffIndex :: [PkgMirrorInfo] -> [PkgMirrorInfo] -> [PkgMirrorInfo] +diffIndex :: [PkgIndexInfo] -> [PkgIndexInfo] -> [PkgIndexInfo] diffIndex as bs = [ pkg | OnlyInLeft pkg <- mergeBy (comparing mirrorPkgId) (sortBy (comparing mirrorPkgId) as) hunk ./MirrorClient.hs 80 (sortBy (comparing mirrorPkgId) bs) ] where - mirrorPkgId (PkgMirrorInfo pkgid _ _ _) = pkgid + mirrorPkgId (PkgIndexInfo pkgid _ _ _) = pkgid hunk ./MirrorClient.hs 82 -subsetIndex :: [PackageId] -> [PkgMirrorInfo] -> [PkgMirrorInfo] +subsetIndex :: [PackageId] -> [PkgIndexInfo] -> [PkgIndexInfo] subsetIndex pkgids = hunk ./MirrorClient.hs 84 - filter (\(PkgMirrorInfo pkgid' _ _ _) -> anyMatchPackage pkgids pkgid') + filter (\(PkgIndexInfo pkgid' _ _ _) -> anyMatchPackage pkgids pkgid') where anyMatchPackage :: [PackageId] -> PackageId -> Bool anyMatchPackage pkgids pkgid' = hunk ./MirrorClient.hs 96 matchPackage pkgid pkgid' = pkgid == pkgid' -mirrorPackages :: Verbosity -> MirrorOpts -> [PkgMirrorInfo] -> HttpSession () +mirrorPackages :: Verbosity -> MirrorOpts -> [PkgIndexInfo] -> HttpSession () mirrorPackages verbosity opts pkgsToMirror = do let credentials = extractCredentials (dstURI opts) hunk ./MirrorClient.hs 117 ok <- downloadFile' src pkgfile when ok $ putPackage dst pkginfo pkgfile - | pkginfo@(PkgMirrorInfo pkgid _ _ _) <- pkgsToMirror ] + | pkginfo@(PkgIndexInfo pkgid _ _ _) <- pkgsToMirror ] where provideAuthInfo :: Maybe (String, String) -> URI -> String -> IO (Maybe (String, String)) hunk ./MirrorClient.hs 136 = Just (username, passwd) extractCredentials _ = Nothing -putPackage :: URI -> PkgMirrorInfo -> FilePath -> HttpSession () -putPackage baseURI (PkgMirrorInfo pkgid mtime muname muid) pkgFile = do +putPackage :: URI -> PkgIndexInfo -> FilePath -> HttpSession () +putPackage baseURI (PkgIndexInfo pkgid mtime muname muid) pkgFile = do putPackageTarball {- case mtime of hunk ./MirrorClient.hs 158 nameStr = display uname -} - ----------------------------------------------------- --- Fetching info from source and destination servers ----------------------------------------------------- - -downloadIndex :: URI -> FilePath -> HttpSession [PkgMirrorInfo] -downloadIndex uri | isOldHackageURI uri = downloadOldIndex uri - | otherwise = downloadNewIndex uri - where - -isOldHackageURI :: URI -> Bool -isOldHackageURI uri - | Just auth <- uriAuthority uri = uriRegName auth == "hackage.haskell.org" - | otherwise = False - - -downloadOldIndex :: URI -> FilePath -> HttpSession [PkgMirrorInfo] -downloadOldIndex uri cacheFile = do - - downloadFile indexURI indexFile - downloadFile logURI logFile - - ioAction $ do - - pkgids <- withFile indexFile ReadMode $ \hnd -> do - content <- BS.hGetContents hnd - case PackageIndex.read (\pkgid _ -> pkgid) (GZip.decompress content) of - Left err -> die $ "Error parsing index at " ++ show uri ++ ": " ++ err - Right pkgs -> return pkgs - - log <- withFile logFile ReadMode $ \hnd -> do - content <- hGetContents hnd - case UploadLog.read content of - Right log -> return log - Left err -> die $ "Error parsing log at " ++ show uri ++ ": " ++ err - - return (mergeLogInfo pkgids log) - - where - indexURI = uri "packages" "archive" "00-index.tar.gz" - indexFile = cacheFile <.> "tar.gz" - - logURI = uri "packages" "archive" "log" - logFile = cacheFile <.> "log" - - mergeLogInfo pkgids log = - catMaybes - . map selectDetails - $ mergeBy (\pkgid entry -> compare pkgid (entryPkgId entry)) - (sort pkgids) - ( map (maximumBy (comparing entryTime)) - . groupBy (equating entryPkgId) - . sortBy (comparing entryPkgId) - $ log ) - - selectDetails (OnlyInRight _) = Nothing - selectDetails (OnlyInLeft pkgid) = - Just $ PkgMirrorInfo pkgid Nothing Nothing Nothing - selectDetails (InBoth pkgid (UploadLog.Entry time uname _)) = - Just $ PkgMirrorInfo pkgid (Just time) (Just uname) Nothing - - entryPkgId (Entry _ _ pkgid) = pkgid - entryTime (Entry time _ _) = time - - -downloadNewIndex :: URI -> FilePath -> HttpSession [PkgMirrorInfo] -downloadNewIndex uri cacheFile = do - downloadFile indexURI indexFile - ioAction $ withFile indexFile ReadMode $ \hnd -> do - content <- BS.hGetContents hnd - case PackageIndex.read selectDetails (GZip.decompress content) of - Left err -> error $ "Error parsing index at " ++ show uri ++ ": " ++ err - Right pkgs -> return pkgs - - where - indexURI = uri "packages/00-index.tar.gz" - indexFile = cacheFile <.> "tar.gz" - - selectDetails :: PackageId -> Tar.Entry -> PkgMirrorInfo - selectDetails pkgid entry = - PkgMirrorInfo - pkgid - (Just time) - (if null username then Nothing else Just (UserName username)) - (if userid == 0 then Nothing else Just (UserId userid)) - where - time = epochTimeToUTC (Tar.entryTime entry) - username = Tar.ownerName (Tar.entryOwnership entry) - userid = Tar.ownerId (Tar.entryOwnership entry) - - epochTimeToUTC :: Tar.EpochTime -> UTCTime - epochTimeToUTC = posixSecondsToUTCTime . realToFrac - - -------------------------- --- HTTP utilities -------------------------- - -infixr 5 - -() :: URI -> FilePath -> URI -uri path = uri { uriPath = Posix.addTrailingPathSeparator (uriPath uri) - Posix. path } - - -type HttpSession a = BrowserAction (HandleStream ByteString) a - -httpSession :: Verbosity -> HttpSession a -> IO a -httpSession verbosity action = - browse $ do - setUserAgent "hackage-mirror" - setErrHandler die - setOutHandler (debug verbosity) - setAllowBasicAuth True - setCheckForProxy True - action - -downloadFile :: URI -> FilePath -> HttpSession () -downloadFile uri file = do - out $ "downloading " ++ show uri ++ " to " ++ file - let etagFile = file <.> "etag" - metag <- ioAction $ catchJustDoesNotExistError - (Just <$> readFile etagFile) - (\_ -> return Nothing) - case metag of - Just etag -> do - let headers = [mkHeader HdrIfNoneMatch (quote etag)] - (_, rsp) <- request (Request uri GET headers BS.empty) - case rspCode rsp of - (3,0,4) -> out $ file ++ " unchanged with ETag " ++ etag - (2,0,0) -> ioAction $ writeDowloadedFileAndEtag rsp - _ -> err (showFailure uri rsp) - - Nothing -> do - (_, rsp) <- request (Request uri GET [] BS.empty) - case rspCode rsp of - (2,0,0) -> ioAction $ writeDowloadedFileAndEtag rsp - _ -> err (showFailure uri rsp) - - where - writeDowloadedFileAndEtag rsp = do - BS.writeFile file (rspBody rsp) - setETag file (unquote <$> findHeader HdrETag rsp) - -getETag :: FilePath -> IO (Maybe String) -getETag file = - catchJustDoesNotExistError - (Just <$> readFile (file ".etag")) - (\_ -> return Nothing) - -setETag :: FilePath -> Maybe String -> IO () -setETag file Nothing = catchJustDoesNotExistError - (removeFile (file <.> "etag")) - (\_ -> return ()) -setETag file (Just etag) = writeFile (file <.> "etag") etag - -catchJustDoesNotExistError = - catchJust (\e -> if isDoesNotExistError e then Just e else Nothing) - -quote s = '"' : s ++ ['"'] - -unquote ('"':s) = go s - where - go [] = [] - go ('"':[]) = [] - go (c:cs) = c : go cs -unquote s = s - --- AAARG! total lack of exception handling in HTTP monad! -downloadFile' :: URI -> FilePath -> HttpSession Bool -downloadFile' uri file = do - out $ "downloading " ++ show uri ++ " to " ++ file - mcontent <- requestGET' uri - case mcontent of - Nothing -> do out $ "404 " ++ show uri - return False - - Just content -> do ioAction $ BS.writeFile file content - return True - -requestGET :: URI -> HttpSession ByteString -requestGET uri = do - (_, rsp) <- request (Request uri GET headers BS.empty) - checkStatus uri rsp - return (rspBody rsp) - where - headers = [] - --- Really annoying! -requestGET' :: URI -> HttpSession (Maybe ByteString) -requestGET' uri = do - (_, rsp) <- request (Request uri GET headers BS.empty) - case rspCode rsp of - (4,0,4) -> return Nothing - _ -> do checkStatus uri rsp - return (Just (rspBody rsp)) - where - headers = [] - - -requestPUT :: URI -> String -> ByteString -> HttpSession () -requestPUT uri mimetype body = do - (_, rsp) <- request (Request uri PUT headers body) - checkStatus uri rsp - where - headers = [ Header HdrContentLength (show (BS.length body)) - , Header HdrContentType mimetype ] - - -checkStatus :: URI -> Response ByteString -> HttpSession () -checkStatus uri rsp = case rspCode rsp of - (2,0,0) -> return () - (4,0,0) -> ioAction (warn normal (showFailure uri rsp)) >> return () - code -> err (showFailure uri rsp) - -showFailure uri rsp = - show (rspCode rsp) ++ " " ++ rspReason rsp ++ show uri - ++ case lookupHeader HdrContentType (rspHeaders rsp) of - Just mimetype | "text/plain" `isPrefixOf` mimetype - -> '\n' : BS.unpack (rspBody rsp) - _ -> "" - ------------------------- -- Command line handling ------------------------- hunk ./MirrorClient.hs 186 "Where to put files during mirroring" ] -validateURI :: String -> Either String URI -validateURI str = case parseURI str of - Nothing -> Left ("invalid URL " ++ str) - Just uri - | uriScheme uri /= "http:" -> Left ("only http URLs are supported " ++ str) - | isNothing (uriAuthority uri) -> Left ("server name required in URL " ++ str) - | otherwise -> Right uri - validateOpts :: [String] -> IO (Verbosity, MirrorOpts) validateOpts args = do let (flags0, args', errs) = getOpt Permute mirrorFlagDescrs args hunk ./MirrorClient.hs 195 when (not (null errs)) (printErrors errs) case args' of - (from:to:pkgstrs) -> case (validateURI from, validateURI to) of + (from:to:pkgstrs) -> case (validateHackageURI from, validateHackageURI to) of (Left err, _) -> die err (_, Left err) -> die err (Right fromURI, Right toURI) -> do hunk ./hackage-server.cabal 181 executable hackage-mirror main-is: MirrorClient.hs other-modules: + Distribution.Client Distribution.Server.LegacyImport.UploadLog Distribution.Server.Users.Types Distribution.Server.Util.Index } [Turn on some warnings for hackage-mirror and fix them Max Bolingbroke **20110927093756 Ignore-this: e03f60dfe9e774da86704e45ed4c125e ] { hunk ./Distribution/Client.hs 208 (\_ -> return ()) setETag file (Just etag) = writeFile (file <.> "etag") etag +catchJustDoesNotExistError :: IO a -> (IOError -> IO a) -> IO a catchJustDoesNotExistError = catchJust (\e -> if isDoesNotExistError e then Just e else Nothing) hunk ./Distribution/Client.hs 212 +quote :: String -> String quote s = '"' : s ++ ['"'] hunk ./Distribution/Client.hs 215 +unquote :: String -> String unquote ('"':s) = go s where go [] = [] hunk ./Distribution/Client.hs 268 checkStatus uri rsp = case rspCode rsp of (2,0,0) -> return () (4,0,0) -> ioAction (warn normal (showFailure uri rsp)) >> return () - code -> err (showFailure uri rsp) + _code -> err (showFailure uri rsp) hunk ./Distribution/Client.hs 270 +showFailure :: URI -> Response ByteString -> String showFailure uri rsp = show (rspCode rsp) ++ " " ++ rspReason rsp ++ show uri ++ case lookupHeader HdrContentType (rspHeaders rsp) of hunk ./MirrorClient.hs 137 extractCredentials _ = Nothing putPackage :: URI -> PkgIndexInfo -> FilePath -> HttpSession () -putPackage baseURI (PkgIndexInfo pkgid mtime muname muid) pkgFile = do +putPackage baseURI (PkgIndexInfo pkgid _mtime _muname _muid) pkgFile = do putPackageTarball {- case mtime of hunk ./hackage-server.cabal 197 --TODO: eliminate: safecopy, cereal, binary, mtl + ghc-options: -Wall + -fno-warn-orphans -fno-warn-unused-do-bind + -fno-warn-warnings-deprecations + -fno-warn-name-shadowing + } [Rough template of Hackage build client Max Bolingbroke **20110927102217 Ignore-this: 7476e699115135d3bef3390e0fef8b8e ] { addfile ./BuildClient.hs hunk ./BuildClient.hs 1 +{-# LANGUAGE PatternGuards #-} +module Main where + +import Network.Browser +import Network.URI (URI(..), URIAuth(..)) + +import Distribution.Client +import Distribution.Server.Util.Merge +import Distribution.Package +import Distribution.Version +import Distribution.Text +import Distribution.Verbosity +import Distribution.Simple.Utils + +import Data.List +import Data.Maybe +import Control.Monad +import qualified Data.ByteString.Lazy.Char8 as BS +import qualified Data.Set as S + +import System.Environment +import System.Exit +import System.FilePath +import System.Directory +import System.Console.GetOpt + + +data BuildOpts = BuildOpts { + srcURI :: URI, + stateDir :: FilePath, + selectedPkgs :: [PackageId] + } + +main :: IO () +main = topHandler $ do + args <- getArgs + (verbosity, opts) <- validateOpts args + + buildOnce verbosity opts + + +buildOnce :: Verbosity -> BuildOpts -> IO () +buildOnce verbosity opts = do + createDirectoryIfMissing False (stateDir opts) + + already_built <- readBuiltCache (stateDir opts) + + built_ok <- httpSession verbosity $ do + index <- downloadIndex (srcURI opts) cacheFile + + let to_build = [pkg_id | PkgIndexInfo pkg_id _ _ _ <- index, pkg_id `S.notMember` already_built] + ioAction $ notice verbosity $ show (length to_build) ++ " packages to build." + + setAuthorityGen $ provideAuthInfo (srcURI opts) $ extractURICredentials (srcURI opts) + filterM (buildPackage verbosity opts) to_build + + writeBuiltCache (stateDir opts) (S.fromList built_ok `S.union` already_built) + where + cacheFile = stateDir opts cacheFileName + cacheFileName | URI { uriAuthority = Just auth } <- srcURI opts + = makeValid (uriRegName auth ++ uriPort auth) + | otherwise + = error $ "unexpected URI " ++ show (srcURI opts) + +readBuiltCache :: FilePath -> IO (S.Set PackageId) +readBuiltCache cache_dir = do + pkgstrs <- liftM lines $ readFile (cache_dir "built") + case validatePackageIds pkgstrs of + Left err -> die err + Right pkgs -> return (S.fromList pkgs) + +writeBuiltCache :: FilePath -> S.Set PackageId -> IO () +writeBuiltCache cache_dir pkgs = writeFile (cache_dir "built") $ unlines $ map show $ S.toList pkgs + + +-- Returns True if we uploaded a build report and documentation for the package successfully +buildPackage :: Verbosity -> BuildOpts -> PackageId -> HttpSession Bool +buildPackage verbosity opts pkg_id = do + out "Not implemented yet!" + return False + + +------------------------- +-- Command line handling +------------------------- + +data BuildFlags = BuildFlags { + flagCacheDir :: Maybe FilePath, + flagVerbosity :: Verbosity, + flagHelp :: Bool +} + +emptyBuildFlags :: BuildFlags +emptyBuildFlags = BuildFlags Nothing normal False + +buildFlagDescrs :: [OptDescr (BuildFlags -> BuildFlags)] +buildFlagDescrs = + [ Option ['h'] ["help"] + (NoArg (\opts -> opts { flagHelp = True })) + "Show this help text" + + , Option ['v'] [] + (NoArg (\opts -> opts { flagVerbosity = moreVerbose (flagVerbosity opts) })) + "Verbose mode (can be listed multiple times e.g. -vv)" + + , Option [] ["cache-dir"] + (ReqArg (\dir opts -> opts { flagCacheDir = Just dir }) "DIR") + "Where to put files during building" + ] + +validateOpts :: [String] -> IO (Verbosity, BuildOpts) +validateOpts args = do + let (flags0, args', errs) = getOpt Permute buildFlagDescrs args + flags = accum flags0 emptyBuildFlags + + when (flagHelp flags) printUsage + when (not (null errs)) (printErrors errs) + + case args' of + (from:pkgstrs) -> case validateHackageURI from of + Left err -> die err + Right fromURI -> do + pkgs <- case validatePackageIds pkgstrs of + Left err -> die err + Right pkgs -> return pkgs + + return $ (,) (flagVerbosity flags) BuildOpts { + srcURI = fromURI, + stateDir = fromMaybe "build-cache" (flagCacheDir flags), + selectedPkgs = pkgs + } + + _ -> do putStrLn "expected a URL: the Hackage server to build for" + printUsage + + where + printUsage = do + putStrLn $ usageInfo usageHeader buildFlagDescrs + exitSuccess + usageHeader = "Usage: hackage-build URL [packages] [options]\nOptions:" + printErrors errs = do + putStrLn $ concat errs ++ "Try --help." + exitFailure + + accum flags = foldr (flip (.)) id flags hunk ./Distribution/Client.hs 15 import Distribution.Package import Distribution.Verbosity import Distribution.Simple.Utils +import Distribution.Text import Data.List import Data.Maybe hunk ./Distribution/Client.hs 47 | isNothing (uriAuthority uri) -> Left ("server name required in URL " ++ str) | otherwise -> Right uri +validatePackageIds :: [String] -> Either String [PackageId] +validatePackageIds pkgstrs = case errs of + (err:_) -> Left $ "'" ++ err ++ "' is not a valid package name or id" + _ -> Right pkgs + where + pkgstrs' = [ (pkgstr, simpleParse pkgstr) | pkgstr <- pkgstrs ] + pkgs = [ pkgid | (_, Just pkgid) <- pkgstrs' ] + errs = [ pkgstr | (pkgstr, Nothing) <- pkgstrs' ] + ---------------------------------------------------- -- Fetching info from source and destination servers ---------------------------------------------------- hunk ./Distribution/Client.hs 62 data PkgIndexInfo = PkgIndexInfo PackageId - (Maybe UTCTime) - (Maybe UserName) - (Maybe UserId) + (Maybe UTCTime) -- Upload time + (Maybe UserName) -- Name of uploader + (Maybe UserId) -- Id of uploader deriving Show downloadIndex :: URI -> FilePath -> HttpSession [PkgIndexInfo] hunk ./Distribution/Client.hs 167 Posix. path } +extractURICredentials :: URI -> Maybe (String, String) +extractURICredentials uri + | Just authority <- uriAuthority uri + , (username, ':':passwd0) <- break (==':') (uriUserInfo authority) + , let passwd = takeWhile (/='@') passwd0 + , not (null username) + , not (null passwd) + = Just (username, passwd) +extractURICredentials _ = Nothing + +provideAuthInfo :: URI -> Maybe (String, String) -> URI -> String -> IO (Maybe (String, String)) +provideAuthInfo for_uri credentials = \uri _realm -> do + if hostName uri == hostName for_uri then return credentials + else return Nothing + where hostName = fmap uriRegName . uriAuthority + + type HttpSession a = BrowserAction (HandleStream ByteString) a httpSession :: Verbosity -> HttpSession a -> IO a hunk ./MirrorClient.hs 99 mirrorPackages :: Verbosity -> MirrorOpts -> [PkgIndexInfo] -> HttpSession () mirrorPackages verbosity opts pkgsToMirror = do - let credentials = extractCredentials (dstURI opts) - setAuthorityGen (provideAuthInfo credentials) + let credentials = extractURICredentials (dstURI opts) + setAuthorityGen (provideAuthInfo (dstURI opts) credentials) sequence_ [ do let srcBasedir hunk ./MirrorClient.hs 119 | pkginfo@(PkgIndexInfo pkgid _ _ _) <- pkgsToMirror ] - where - provideAuthInfo :: Maybe (String, String) -> URI -> String -> IO (Maybe (String, String)) - provideAuthInfo credentials = \uri _realm -> do - if hostName uri == hostName (dstURI opts) then return credentials - else return Nothing - - hostName = fmap uriRegName . uriAuthority - - extractCredentials uri - | Just authority <- uriAuthority uri - , (username, ':':passwd0) <- break (==':') (uriUserInfo authority) - , let passwd = takeWhile (/='@') passwd0 - , not (null username) - , not (null passwd) - = Just (username, passwd) - extractCredentials _ = Nothing - putPackage :: URI -> PkgIndexInfo -> FilePath -> HttpSession () putPackage baseURI (PkgIndexInfo pkgid _mtime _muname _muid) pkgFile = do putPackageTarball hunk ./MirrorClient.hs 182 (Left err, _) -> die err (_, Left err) -> die err (Right fromURI, Right toURI) -> do - let pkgstrs' = [ (pkgstr, simpleParse pkgstr) | pkgstr <- pkgstrs ] - pkgs = [ pkgid | (_, Just pkgid) <- pkgstrs' ] - errs = [ pkgstr | (pkgstr, Nothing) <- pkgstrs' ] - case errs of - (err:_) -> die $ "'" ++ err ++ "' is not a valid package name or id" - _ -> return () + pkgs <- case validatePackageIds pkgstrs of + Left err -> die err + Right pkgs -> return pkgs return $ (,) (flagVerbosity flags) MirrorOpts { srcURI = fromURI, hunk ./hackage-server.cabal 202 -fno-warn-warnings-deprecations -fno-warn-name-shadowing +executable hackage-build + main-is: BuildClient.hs + other-modules: + Distribution.Client + Distribution.Server.LegacyImport.UploadLog + Distribution.Server.Users.Types + Distribution.Server.Util.Index + Distribution.Server.Util.Merge + build-depends: + base, + containers, bytestring, pretty, + filepath, directory, + time, old-locale, + tar, zlib, + network, HTTP, + Cabal, + --TODO: eliminate: + safecopy, cereal, binary, mtl + + ghc-options: -Wall + -fno-warn-orphans -fno-warn-unused-do-bind + -fno-warn-warnings-deprecations + -fno-warn-name-shadowing } [Bump acid-state and happstack versions to avoid bug Max Bolingbroke **20110928122814 Ignore-this: 67da02f641e393571db92286e06fd23a There is a bug in happstack-server before v6.2.2 when you try to read the cookies/request params of a POST/PUT request with a content-type different from multipart/form-data or application/x-www-form-urlencoded The bug is described in http://comments.gmane.org/gmane.comp.lang.haskell.web/2033 As a result I changed the version bounds to v6.2.* ] hunk ./hackage-server.cabal 162 rss == 3000.1.*, Cabal == 1.10.*, csv == 0.1.*, - acid-state == 0.4.*, - happstack-server == 6.1.*, + acid-state == 0.5.*, + happstack-server == 6.2.*, happstack-util == 6.0.* if ! flag(minimal) [A mostly functional build client Max Bolingbroke **20110928211352 Ignore-this: d65dd5e7e53015635d97ab3f9a38b298 ] { hunk ./BuildClient.hs 8 import Network.URI (URI(..), URIAuth(..)) import Distribution.Client -import Distribution.Server.Util.Merge import Distribution.Package hunk ./BuildClient.hs 9 -import Distribution.Version import Distribution.Text import Distribution.Verbosity hunk ./BuildClient.hs 11 -import Distribution.Simple.Utils +import Distribution.Simple.Utils hiding (intercalate) import Data.List import Data.Maybe hunk ./BuildClient.hs 15 +import Data.IORef +import Control.Exception import Control.Monad import qualified Data.ByteString.Lazy.Char8 as BS import qualified Data.Set as S hunk ./BuildClient.hs 21 +import qualified Codec.Compression.GZip as GZip +import qualified Codec.Archive.Tar as Tar + import System.Environment import System.Exit import System.FilePath hunk ./BuildClient.hs 29 import System.Directory import System.Console.GetOpt +import System.Process +import System.IO.Error data BuildOpts = BuildOpts { hunk ./BuildClient.hs 35 srcURI :: URI, + username :: String, + password :: String, stateDir :: FilePath, selectedPkgs :: [PackageId] } hunk ./BuildClient.hs 41 +srcName :: BuildOpts -> String +srcName opts = fromMaybe (show (srcURI opts)) (uriHostName (srcURI opts)) + + main :: IO () main = topHandler $ do args <- getArgs hunk ./BuildClient.hs 55 buildOnce :: Verbosity -> BuildOpts -> IO () buildOnce verbosity opts = do - createDirectoryIfMissing False (stateDir opts) + (does_not_have_docs, mark_as_having_docs, persist) <- mkPackageHasDocs opts hunk ./BuildClient.hs 57 - already_built <- readBuiltCache (stateDir opts) + prepareBuildPackages verbosity opts + flip finally persist $ httpSession verbosity $ do + -- Make sure we authenticate to Hackage + setAuthorityGen $ provideAuthInfo (srcURI opts) $ Just (username opts, password opts) hunk ./BuildClient.hs 62 - built_ok <- httpSession verbosity $ do + -- Find those files *not* marked as having documentation in our cache index <- downloadIndex (srcURI opts) cacheFile hunk ./BuildClient.hs 64 + to_build <- filterM does_not_have_docs [pkg_id | PkgIndexInfo pkg_id _ _ _ <- index + , null (selectedPkgs opts) || pkg_id `elem` selectedPkgs opts] + ioAction $ notice verbosity $ show (length to_build) ++ " packages to build documentation for." hunk ./BuildClient.hs 68 - let to_build = [pkg_id | PkgIndexInfo pkg_id _ _ _ <- index, pkg_id `S.notMember` already_built] - ioAction $ notice verbosity $ show (length to_build) ++ " packages to build." - - setAuthorityGen $ provideAuthInfo (srcURI opts) $ extractURICredentials (srcURI opts) - filterM (buildPackage verbosity opts) to_build - - writeBuiltCache (stateDir opts) (S.fromList built_ok `S.union` already_built) + -- Try to build each of them, uploading the documentation and build reports + -- along the way. We mark each package as having documentation in the cache even + -- if the build fails because we don't want to keep continually trying to build + -- a failing package! + forM_ to_build $ \pkg_id -> do + buildPackage verbosity opts pkg_id + ioAction $ mark_as_having_docs pkg_id where cacheFile = stateDir opts cacheFileName cacheFileName | URI { uriAuthority = Just auth } <- srcURI opts hunk ./BuildClient.hs 82 | otherwise = error $ "unexpected URI " ++ show (srcURI opts) -readBuiltCache :: FilePath -> IO (S.Set PackageId) -readBuiltCache cache_dir = do - pkgstrs <- liftM lines $ readFile (cache_dir "built") - case validatePackageIds pkgstrs of - Left err -> die err - Right pkgs -> return (S.fromList pkgs) +-- Builds a little memoised function that can tell us whether a particular package already has documentation +mkPackageHasDocs :: BuildOpts -> IO (PackageId -> HttpSession Bool, PackageId -> IO (), IO ()) +mkPackageHasDocs opts = do + init_already_built <- readBuiltCache (stateDir opts) + cache_var <- newIORef init_already_built + + let mark_as_having_docs pkg_id = atomicModifyIORef cache_var $ \already_built -> (S.insert pkg_id already_built, ()) + does_not_have_docs pkg_id = do + has_docs <- ioAction $ liftM (pkg_id `S.member`) $ readIORef cache_var + if has_docs + then return False + else do + has_no_docs <- liftM isNothing $ requestGET' (srcURI opts "package" display pkg_id "doc") + unless has_no_docs $ ioAction $ mark_as_having_docs pkg_id + return has_no_docs + persist = readIORef cache_var >>= writeBuiltCache (stateDir opts) + + return (does_not_have_docs, mark_as_having_docs, persist) + where + readBuiltCache :: FilePath -> IO (S.Set PackageId) + readBuiltCache cache_dir = do + pkgstrs <- handleDoesNotExist (return []) $ liftM lines $ readFile (cache_dir "built") + case validatePackageIds pkgstrs of + Left err -> die err + Right pkgs -> return (S.fromList pkgs) + + writeBuiltCache :: FilePath -> S.Set PackageId -> IO () + writeBuiltCache cache_dir pkgs = writeFile (cache_dir "built") $ unlines $ map display $ S.toList pkgs + hunk ./BuildClient.hs 112 -writeBuiltCache :: FilePath -> S.Set PackageId -> IO () -writeBuiltCache cache_dir pkgs = writeFile (cache_dir "built") $ unlines $ map show $ S.toList pkgs +prepareBuildPackages :: Verbosity -> BuildOpts -> IO () +prepareBuildPackages verbosity opts = withCurrentDirectory (stateDir opts) $ do + writeFile "config" $ unlines [ + "remote-repo: " ++ srcName opts ++ ":" ++ show (srcURI opts "packages" "archive"), + "remote-repo-cache: " ++ stateDir opts "packages", + "library-for-ghci: False", + "package-db: " ++ stateDir opts "local.conf.d", + "documentation: True", + "remote-build-reporting: detailed", + -- Used for the "upload" commands only, not "report" :-( + "username: " ++ username opts, + "password: " ++ password opts + ] hunk ./BuildClient.hs 126 + -- Create cache for the empty configuration directory + local_conf_d_exists <- doesDirectoryExist "local.conf.d" + unless local_conf_d_exists $ do + createDirectory "local.conf.d" + _ <- runProcess "ghc-pkg" ["recache", "--package-conf=" ++ stateDir opts "local.conf.d"] + Nothing Nothing Nothing Nothing Nothing >>= waitForProcess + return () + + update_ec <- cabal verbosity opts "update" [] + unless (update_ec == ExitSuccess) $ die "Could not 'cabal update' from specified server" hunk ./BuildClient.hs 137 --- Returns True if we uploaded a build report and documentation for the package successfully -buildPackage :: Verbosity -> BuildOpts -> PackageId -> HttpSession Bool + +buildPackage :: Verbosity -> BuildOpts -> PackageId -> HttpSession () buildPackage verbosity opts pkg_id = do hunk ./BuildClient.hs 140 - out "Not implemented yet!" - return False + -- The documentation is installed within the stateDir because we set a prefix while installing + let doc_dir = stateDir opts "share" "doc" display pkg_id "html" + temp_doc_dir = stateDir opts "share" "doc" display pkg_id display (pkgName pkg_id) + + mb_docs <- ioAction $ withCurrentDirectory (stateDir opts) $ do + -- Let's not clean in between installations. This should save us some download/installation time for packages + -- that are depended on by a lot of things, at the cost of some disk space. + --handleDoesNotExist (return ()) $ removeDirectoryRecursive "packages" + --handleDoesNotExist (return ()) (removeDirectoryRecursive "local.conf.d") >> createDirectoryIfMissing False "local.conf.d" + + -- We CANNOT build from an unpacked directory, because Cabal only generates build reports + -- if you are building from a tarball that was verifiably downloaded from the server + cabal verbosity opts "install" ["--enable-documentation", "--enable-tests", + "--html-location=" ++ show (srcURI opts "package" "$pkg-$version" "doc"), + "--prefix=" ++ stateDir opts, display pkg_id] + + -- Submit a report even if installation/tests failed: all interesting data points! + report_ec <- cabal verbosity opts "report" ["--username", username opts, "--password", password opts] + + -- Delete reports after submission because we don't want to submit them *again* in the future + when (report_ec == ExitSuccess) $ do + dotCabal <- getAppUserDataDirectory "cabal" + removeDirectoryRecursive (dotCabal "reports" srcName opts) + + docs_generated <- doesDirectoryExist doc_dir + if docs_generated + then do + notice verbosity $ "Docs generated for " ++ display pkg_id + liftM Just $ + -- We need the files in the documentation .tar.gz to have paths like foo/index.html + -- Unfortunately, on disk they have paths like foo-x.y.z/html/index.html. This hack resolves the problem: + bracket (renameDirectory doc_dir temp_doc_dir) (\() -> renameDirectory temp_doc_dir doc_dir) $ \() -> + tarGzDirectory temp_doc_dir + else return Nothing + + -- Submit the generated docs, if possible + case mb_docs of + Nothing -> return () + Just docs_tgz -> requestPUT (srcURI opts "package" display pkg_id "doc") "application/x-gzip" docs_tgz + +cabal :: Verbosity -> BuildOpts -> String -> [String] -> IO ExitCode +cabal verbosity opts cmd args = do + cwd <- getCurrentDirectory + let all_args = ("--config-file=" ++ stateDir opts "config"):cmd:args + notice verbosity $ "cd " ++ cwd ++ " && cabal " ++ intercalate " " all_args + ph <- runProcess "cabal" all_args (Just cwd) Nothing Nothing Nothing Nothing + waitForProcess ph + +tarGzDirectory :: FilePath -> IO BS.ByteString +tarGzDirectory dir = do + --print (containing_dir, nested_dir) + res <- liftM (GZip.compress . Tar.write) $ Tar.pack containing_dir [nested_dir] + BS.writeFile "/tmp/foo.tar.gz" res + --BS.length res `seq` print "tarGzDirectory" + return res + where (containing_dir, nested_dir) = splitFileName dir + +withCurrentDirectory :: FilePath -> IO a -> IO a +withCurrentDirectory cwd' act = bracket (do { cwd <- getCurrentDirectory; setCurrentDirectory cwd'; return cwd }) + setCurrentDirectory (const act) ------------------------- hunk ./BuildClient.hs 246 Left err -> die err Right pkgs -> return pkgs + (usrnme, psswrd) <- case extractURICredentials fromURI of + Nothing -> die "No Hackage server username/password" + Just creds -> return creds + + -- Ensure we store the absolute state_dir, because we might change the CWD later + -- and we don't want the stateDir to be invalidated by such a change + -- + -- We have to ensure the directory exists before we do canonicalizePath, or + -- otherwise we get an exception if it does not yet exist + let rel_state_dir = fromMaybe "build-cache" (flagCacheDir flags) + createDirectoryIfMissing False rel_state_dir + abs_state_dir <- canonicalizePath rel_state_dir + return $ (,) (flagVerbosity flags) BuildOpts { hunk ./BuildClient.hs 260 - srcURI = fromURI, - stateDir = fromMaybe "build-cache" (flagCacheDir flags), + srcURI = removeURICredentials fromURI, + username = usrnme, + password = psswrd, + stateDir = abs_state_dir, selectedPkgs = pkgs } hunk ./BuildClient.hs 280 exitFailure accum flags = foldr (flip (.)) id flags + + +handleDoesNotExist :: IO a -> IO a -> IO a +handleDoesNotExist handler act = handleJust (\e -> if isDoesNotExistError e then Just () else Nothing) (\() -> handler) act hunk ./Distribution/Client.hs 177 = Just (username, passwd) extractURICredentials _ = Nothing +removeURICredentials :: URI -> URI +removeURICredentials uri = uri { uriAuthority = fmap (\auth -> auth { uriUserInfo = "" }) (uriAuthority uri) } + provideAuthInfo :: URI -> Maybe (String, String) -> URI -> String -> IO (Maybe (String, String)) provideAuthInfo for_uri credentials = \uri _realm -> do hunk ./Distribution/Client.hs 182 - if hostName uri == hostName for_uri then return credentials - else return Nothing - where hostName = fmap uriRegName . uriAuthority + if uriHostName uri == uriHostName for_uri then return credentials + else return Nothing + +uriHostName :: URI -> Maybe String +uriHostName = fmap uriRegName . uriAuthority type HttpSession a = BrowserAction (HandleStream ByteString) a hunk ./Distribution/Client.hs 287 headers = [] -requestPUT :: URI -> String -> ByteString -> HttpSession () -requestPUT uri mimetype body = do - (_, rsp) <- request (Request uri PUT headers body) +requestPUTFile :: URI -> String -> FilePath -> HttpSession () +requestPUTFile uri mime_type file = do + content <- ioAction $ BS.readFile file + requestPUT uri mime_type content + +requestPOST, requestPUT :: URI -> String -> ByteString -> HttpSession () +requestPOST = requestPOSTPUT POST +requestPUT = requestPOSTPUT PUT + +requestPOSTPUT :: RequestMethod -> URI -> String -> ByteString -> HttpSession () +requestPOSTPUT meth uri mimetype body = do + (_, rsp) <- request (Request uri meth headers body) checkStatus uri rsp where headers = [ Header HdrContentLength (show (BS.length body)) hunk ./MirrorClient.hs 18 import Data.List import Data.Maybe import Control.Monad -import qualified Data.ByteString.Lazy.Char8 as BS import System.Environment import System.Exit hunk ./MirrorClient.hs 126 Nothing -> return () Just time -> putPackageUploadTime -} where - putPackageTarball = do - pkgContent <- ioAction $ BS.readFile pkgFile - let pkgURI = baseURI display pkgid <.> "tar.gz" - requestPUT pkgURI "application/x-gzip" pkgContent + pkgURI = baseURI display pkgid <.> "tar.gz" + putPackageTarball = requestPUTFile pkgURI "application/x-gzip" pkgFile {- putPackageUploadTime time = do (_, rsp) <- request (requestPUT pkgURI "text/plain" timeStr) hunk ./hackage-server.cabal 160 json == 0.4.*, stringsearch == 0.3.*, rss == 3000.1.*, - Cabal == 1.10.*, + Cabal == 1.11.*, csv == 0.1.*, acid-state == 0.5.*, happstack-server == 6.2.*, hunk ./hackage-server.cabal 213 build-depends: base, containers, bytestring, pretty, - filepath, directory, + filepath, directory, process, time, old-locale, tar, zlib, network, HTTP, } [Comments/debug code only Max Bolingbroke **20110928211754 Ignore-this: 997aeb8b78d431180c90dfd7f47e42af ] { hunk ./BuildClient.hs 33 import System.IO.Error +-- FIXMEs: +-- * Sandbox user-supplied code (i.e. the whole 'cabal install' invocation) + + data BuildOpts = BuildOpts { srcURI :: URI, username :: String, hunk ./BuildClient.hs 194 tarGzDirectory :: FilePath -> IO BS.ByteString tarGzDirectory dir = do - --print (containing_dir, nested_dir) res <- liftM (GZip.compress . Tar.write) $ Tar.pack containing_dir [nested_dir] hunk ./BuildClient.hs 195 - BS.writeFile "/tmp/foo.tar.gz" res - --BS.length res `seq` print "tarGzDirectory" return res where (containing_dir, nested_dir) = splitFileName dir } Context: [Let users know about the default admin account Duncan Coutts **20110925073012 Ignore-this: cc64e6efd66e0f69bc0eaf98ae99e4b0 ] [Improve message about server running Duncan Coutts **20110925072943 Ignore-this: 6c1dbbdaba2170492ddf7fbf7df8fd98 ] [Add help instructions to make new empty server Duncan Coutts **20110925072918 Ignore-this: 7932e6c4de2ee73252e4cf46e1fe452a ] [Fix UploadMonad for mtl-2 Duncan Coutts **20110815232137 Ignore-this: 30fd1bceebc5de9b21eb9f6e1c76a790 ] [Remove some done TODOs Duncan Coutts **20110815204223 Ignore-this: b8313f0092ee54b426ba3a91a9087d53 ] [Fix build warnings Duncan Coutts **20110815204025 Ignore-this: 4b22faa70ad1ae7664d5c09b6214959d ] [added basic functionality for viewing changelogs Stefan Wehr **20110815125222 Ignore-this: 751929efdec981848f4ff167eceb9c60 ] [Rename auth modules into the framework Duncan Coutts **20110815201045 Ignore-this: b9a1068c882ce4845b0ae1083b01b4bb ] [Clean up the HTTP basic and digest implementation Duncan Coutts **20110815195156 Ignore-this: 44aff4742f043e348258dd67fa1d685a ] [Remove the per-user auth type from the users dump/restore Duncan Coutts **20110815191218 Ignore-this: f068a29dbb27a3ed72d136a18c9b6837 ] [Move restore --tarball option to an argument Ben Millwood **20110814111551 Ignore-this: 9cb86774aa66683a8fd12d6495c8bcc2 The tarball specification isn't optional, so it's somewhat counterintuitive to make it an option. ] [Check for the static dir only when necessary Ben Millwood **20110814100259 Ignore-this: b1408a5afb22cb97a625c4ec288e3f2c We originally checked for the static dir whenever the server was initialised, even if it was only running for a backup or restore. It turned out this check is already done per-command in Main anyway, but just to be careful I moved the check from initialisation to running, where it's actually relevant. ] [Fix typo Successly -> Successfully Ben Millwood **20110813171415 Ignore-this: 5674367a3f79770ddb2b4d00c533a65c ] [fixed small typo (missing blank) Stefan Wehr **20110814110306 Ignore-this: 2fe4938b96f8f0f11ac75da63a67774c ] [added rudimentary admin page Stefan Wehr **20110814110240 Ignore-this: 3280b81268ef460e2997dda89e8181c8 ] [Remove ability to set per-user basic/digest auth method Duncan Coutts **20110814144826 Ignore-this: f083db266d36a81724dc6f4c671ba1b0 ] [Remove force auth type from withHackageAuth Duncan Coutts **20110814144036 Ignore-this: 5ea30b8ed126b26c45f19ecaa53fade7 And always offer both basic and digest auth methods ] [Remove the per-user basic/digest auth switch Duncan Coutts **20110814142324 Ignore-this: 6d32929ebd3e8a037b4e192f5c90c694 ] [Remove the old crypt based basic password storage Duncan Coutts **20110814112636 Ignore-this: 30ede85ada072a057c1594aa52fc74e ] [Switch HtPasswdDb to be just for legacy import Duncan Coutts **20110814112052 Ignore-this: e4a6dd0026ebfe6fe5f2f384d5654e44 We cannot import them as using the same password digest as other accounts. ] [Change the HackageFeature interface to put backup dump and restore together Duncan Coutts **20110813154348 Ignore-this: 8a276c5e4a777f731bd590c481ced424 ] [Rename to initHackageFeatures Duncan Coutts **20110813151913 Ignore-this: 4f8262535e6ced33487f559f814ca079 ] [Don't need ExistentialQuantification in Distribution.Server.Features anymore Duncan Coutts **20110813151839 Ignore-this: b19b78e679a00bf1746b9461788e2691 ] [Stop passing the blob storage to dump/restoreBackup Duncan Coutts **20110813105607 Ignore-this: fb2512db8bba1dc81b6051cdeac317e0 Features that need it can use it themselves since it's part of the ServerEnv that they are initialised with. ] [Remove digest authentication qop=auth-int support Duncan Coutts **20110813105414 Ignore-this: e049e49aa82c8fae715c98bce2189461 The fact that it needs to consume the whole HTTP message body makes it problematic. We can perhaps reintroduce it later to secure uploading of package tarballs. ] [Move the feature initHooks from the class into the HackageFeature record Duncan Coutts **20110807232033 Ignore-this: 2a97cc172c6f8d4a78f38bda8c6a8be3 ] [Rename the HackageModule/Feature type and some members Duncan Coutts **20110807222709 Ignore-this: 12950620308852e004706f06099e5165 ] [Fix <> on init Duncan Coutts **20110812140717 Ignore-this: 174499c5ba3ba65d5cec609d29cf85a4 ] [Rename the server Config to ServerEnv to reduce confusion Duncan Coutts **20110807203337 Ignore-this: 177124869e8dcb6368eedd4c3602c76d ] [Move to Cabal-1.8 rules for component deps Duncan Coutts **20110807103954 Ignore-this: 33eeb44a6a1fe84a465f9ae4e24b75e5 ] [Add a cabal flag for a build configuration with a minimal featureset Duncan Coutts **20110806113612 Ignore-this: 9e9c46dc434a33b7545e6d9909a8e42d ] [Add the missing Framework module Duncan Coutts **20110806111231 Ignore-this: 9a1c9474edd077a0d451d19635da3b33 ] [Tidy up the .cabal file Duncan Coutts **20110729110840 Ignore-this: 5d71117dbad46b249baf0f5b62097bae ] [Update author, maintainer and copyright info in .cabal file Duncan Coutts **20110729105124 Ignore-this: c0d05aebd6954a7af85b3bda0e1ecc0e ] [Remove unused dependency on the uri package Duncan Coutts **20110729104932 Ignore-this: 937aa41b93536d2c5dec696cdb39b756 ] [Handle old hackage URLs in the mirror client Duncan Coutts **20110725023021 Ignore-this: 601eb58403a7a1d6b74ef0d6ba786e8f ] [Support conditional/cached downloads Duncan Coutts **20110725014851 Ignore-this: 790049ba617d69379c51ab457d66a73b Requires another patch to the HTTP lib ] [Stop exporting various internals of features Duncan Coutts **20110725004217 Ignore-this: 588ae6524965706bbcb44a73447a0609 ] [Move and rename a number of modules Duncan Coutts **20110724202855 Ignore-this: 892c67234f999d3ed29cfd964a5121c8 Trying to get a handle on what goes together logically and what the module dependencies are. ] [Warn but do not fail on packages that are rejected Duncan Coutts **20110724184114 Ignore-this: 2af4c4176a2d79de58203a3135f1f4ec ] [Use digest auth by default Duncan Coutts **20110724183829 Ignore-this: b607227f21d8a67bf01f2e5dea6a4d81 ] [Fix PackageList not recognizing newly uploaded packages Matthew Gruen **20110724022422 Ignore-this: edac02156481c0b9b5f25dbe54938f9 ] [Correct inconsistency in auth realm Matthew Gruen **20110723224940 Ignore-this: 7790a4a19af198a8fc5febff806bcc6f Realms do likely need more customization/coordination ] [Link to the user registration page from the User accounts page. David Lazar **20110723223630 Ignore-this: 8a2b280e6911aecaadae19339b7c9ff7 ] [Rework the Mirror upload feature Duncan Coutts **20110723220846 Ignore-this: d251e00d329db4e342a7e03a3643cb22 So it now does not use multi-part format, just PUT the raw .cabal or .tar.gz file directly. ] [Clean up Tags related cruft due to an inattentive developer Thomas.DuBuisson@gmail.com**20110723214829 Ignore-this: 740ec1302b026c6a594ebd89e1a19687 ] [Calculate the immutable tags on package upload Thomas.DuBuisson@gmail.com**20110723212556 Ignore-this: 71d495a7dbf48174f18ae448ac07bb4 ] [Prevent packages from being deprecated in favor of themselves. David Lazar **20110723195532 Ignore-this: 12fd7511b1e486dd54e8cc9be1884dd5 ] [Allow listing specific packages to mirror on the command line Duncan Coutts **20110723211804 Ignore-this: 5cf915250f68b74786f842251905f3bf ] [Fix the 401 not authorised response in the basic & digest auth code Duncan Coutts **20110723165829 Ignore-this: 10d776ccffa2aeaa99a3f285372b4665 ] [Fix to the user group editor resource: fix removal of users Duncan Coutts **20110723102851 Ignore-this: 84df907391174af1fd500f046d4c829f The groupResourceAt function makes a user group into a resource. It was not handling the remove correctly. ] [Add consumeRequestBody utility and use it to replace takeRequestBody Duncan Coutts **20110723094914 Ignore-this: a380baaad71e7236b075e18609c65476 It simplifies the error handling. ] [Add a TODO about clean shutdowns Duncan Coutts **20110723094844 Ignore-this: 942ce02c27c447188250df0b59e6da7e ] [Clarify the top level quota stuff and the http method hack Duncan Coutts **20110723094742 Ignore-this: c4c671221a799a3b63a2b712678a2d3d ] [Fix build error, probably from the Either/Error patch Thomas.DuBuisson@gmail.com**20110723041723 Ignore-this: b5eb91ea00e0d139d0a92042454ab4d6 ] [Change how we handle exceptions that result in error pages Duncan Coutts **20110723001635 Ignore-this: 7ebf7080445454ec9e252f8529e3dd0f Instead of returning (Either ErrorResponse a) everywhere, use an error monad layered on top of ServerPartT. So now we can throw ErrorResponse as exceptions and not clutter up the code with error checking. We can also still do nicely formatted html error pages by using an exception handler. ] [Improve deprecation form. David Lazar **20110723001614 Ignore-this: ebcd1fff01fecf29366108f024f959c3 ] [Fix the tags edit capability Thomas.DuBuisson@gmail.com**20110722234623 Ignore-this: e0ed903d62b6f067ab4ddb0a05afe45f ] [Fix mungeRequest (thanks to Jeremy). David Lazar **20110722222457 Ignore-this: 72483da60ccf900e5fcb6ae66652e71d ] [added missing Distribution.Server.Acid Jeremy Shaw **20110722212358 Ignore-this: 1692c0b788ec2f0efd697e8c3c0c3772 ] [migrate from happstack-data/happstack-state to safecopy/acid-state Jeremy Shaw **20110722204241 Ignore-this: 9daef014eb49d9efeeb337960016e6ad ] [Add missing word in upload notes. David Lazar **20110722191719 Ignore-this: 8b84bd4104069b7fb92c78bf53cacec2 ] [Delete todo comment about hardcoded /tmp. David Lazar **20110722191527 Ignore-this: 5a8c59628c6c5aaf107e65b041bb5592 ] [Fix minor layout issue. David Lazar **20110722184711 Ignore-this: b28272a0634254d47f4af55966b9a8fb ] [Un-hardcode the "/tmp" directory. David Lazar **20110722183315 Ignore-this: d9be80c40935d2407e3c3cfdd78495b9 ] [Fix type error in Reports module Duncan Coutts **20110722160402 Ignore-this: e4af86faf06acfd2e575e4d2438ed234 ] [Record in the TODO that the error handling is horrible Duncan Coutts **20110721145719 Ignore-this: 34412b930e8e8126803daf93508e5b40 ] [Use takeRequestBody rather than the old rqBody Duncan Coutts **20110721144933 Ignore-this: 61a1190e5287afd9dbb565483db65953 The error handling here is really terrible, extremely verbose. ] [Build report POST handler works in two steps Vo Minh Thu **20101108145443 Ignore-this: 502ee2cd976a035fb2bde2737e848943 The code for handling the second step separately was already existing. Modification amounts to look for the data in the POST body instead as a field and removing the lookup for the log. ] [Require Cabal-1.10 only Duncan Coutts **20110721134641 Ignore-this: dc4db0dfa3a9c26fbe342e81201a9f48 ] [Use file globs for data-files in the .cabal file Duncan Coutts **20110721134629 Ignore-this: a3e98d0bbf3d27aab3b9e43ab96ce64d ] [Rewrite the mirroring client Duncan Coutts **20110721134411 Ignore-this: 559ebb63a87f0d5dd3ad131857d4924b ] [Rewrite the command line handling and toplevel/main code Duncan Coutts **20110721134125 Ignore-this: 3e2d94bf9067b109b5944576007f7c33 Also improve the handling of the tmp dir. ] [Relax a couple dependencies to allow building with a standard ghc-6.12 setup Duncan Coutts **20110721132121 Ignore-this: a8c357c98e5c0b3713493c8c0e93284d ] [partial upgrade to happstack-server 6.*. Look for HS6 markers in code to find remaining work. Jeremy Shaw **20110720172641 Ignore-this: 531988f51b18e8bd204ff36a6e82158d ] [add crypt to executable hackage-mirror too Jens Petersen **20101204071019 Ignore-this: a91cc08ee141c192e6b01db5704b86c2 ] [port over new package page from hackage-scripts Antoine Latter **20101122052249 Ignore-this: 7c3a9e5bbf9abd5cb2c7358166083453 ] [-Wall cleaner Antoine Latter **20101121190634 Ignore-this: 344310c40862d0ab8915ba18b7b5fd43 ] [redirect urls of the form /packages/archive/$pkg/latest/... properly Antoine Latter **20101121175330 Ignore-this: 3f19b43349cc03df656df6d35a0c02db ] [rollback part of 'changes to constructing the index tarball' Antoine Latter **20101121061703 Ignore-this: f7fdd439a7dce12a01bbce3f0cf4d455 ] [redirect documentation links with no version to the most recent version Antoine Latter **20101119050537 Ignore-this: 6584789f6f8637e5a7cd79a778e87bbe future work: redirect to the most recent version with documentation ] [changes to constructing the index tarball Antoine Latter **20101119010644 Ignore-this: cad4d1bcd791893e55f48fc524688640 + tar paths are always posix + match wacky format in current index by prefixing package descriptions with './'. This lets us send side-band data non prefixed. ] [bump dependencies in package description Antoine Latter **20101118032016 Ignore-this: 3abe5e1f3e50b9cea970051eb6e11784 'time' to 1.2 'Cabal' to 1.8 or 1.10 (1.10 is not yet on Hackage) ] [Redirecting a POST is tricky, so instead we re-host the upload handler at various URLs. Antoine Latter **20101106223458 Ignore-this: 1b631fb91ce4b64ca61db5adffe1df18 ] [We have to types of legacy URLs to support: hackage.haskell.org and non-hackage URLs Antoine Latter **20101106161232 Ignore-this: 96d400eccb2d0ecae6ba39ef764dd060 This adds support for /upload to redirect, in addition to /packages/upload. ] [the authorization realm used by cabal-install is "Hackage", not "hackage" Antoine Latter **20101106160954 Ignore-this: 5249289823cc857a112dbc5f9e549726 ] [Add --static-dir flag for the command new Duncan Coutts **20101106150734 Ignore-this: 9ad4bdb1ac0d82cb90448dded25a9dd6 ] [Support Bulk update of distro info using CSV format Joachim Breitner **20101106125952 Ignore-this: dbac46e3f798e703de33a5b237172a8b ] [Create a new ResourceType CSVFile Joachim Breitner **20101106125928 Ignore-this: 385cd314ae887ef679e514907974bdfe ] [Minor changes to URI and HTTP response code handling in mirror client Duncan Coutts **20101106142927 Ignore-this: 9e90f998a50e339a8b3768852d780684 ] [Improve import, add bits of platform feature Matthew Gruen **20101010194336 Ignore-this: e3f6554e8e8bc176898b3a62a6f5fbe Bulk import fully working with historical accounts Tags and maintainers have full backup ] [Add executable hackage-mirror to cabal file Matthew Gruen **20101003144329 Ignore-this: 36dac6380ea64e0cdb54fb384956ce0d ] [GHC 7 no longer supports generallizing types in a 'where' clause. Antoine Latter **20100926014024 Ignore-this: 3d0b73738deced799a28e5f64690c338 ] [Update backup modules Matthew Gruen **20100822052740 Ignore-this: 7f995f174163b3f3ad784e1a235fb13b Move to more consistent naming (shared *.Backup structure) Prepare to implement backup for additional features ] [Duplicate index in reverse dependencies Matthew Gruen **20100822004120 Ignore-this: c91efbbb6cdf94c0ce859417a9b00e68 Otherwise, large update arguments are required, which makes large event files ] [Update TODO file Matthew Gruen **20100816180627 Ignore-this: 958bc720cffc2f6a31234a637b1b3be3 Clear out implemented items Add some directions for server development to go in ] [Some polish; additional comments Matthew Gruen **20100816174827 Ignore-this: e1437d38236d7145fbd87049ade1e472 ] [Integrate mirror feature, with client that diffs hackage-scripts log and hackage-server index Matthew Gruen **20100813003348 Ignore-this: 788b66c1a12771f60627f693ec27dfd8 ] [Add preferred-versions to index tarball and move HTML index pages to the html feature Matthew Gruen **20100813003049 Ignore-this: c716f4a446f3743cca035739fe34ddb4 ] [Modify user index to explicitly incorporate historical data (breaks happstack-state data); improve the internal interface for user groups by adding groupExists Matthew Gruen **20100813003017 Ignore-this: 6761c9a487b91bafcdeb792b6da31063 ] [Get rid of 503 as static web page, insert HTML directly into source Matthew Gruen **20100808090458 Ignore-this: 245b0001bb953eb71534e08fb82b73a7 ] [Have temp server while data is loading Matthew Gruen **20100808070325 Ignore-this: d94522632fc17c84e978dc9162b9dfbd ] [Add some time logging, for sparky metrics Matthew Gruen **20100808003533 Ignore-this: da7509b855655774cdaeafcb8a6f1ad1 ] [Replace parallel with deepseq Matthew Gruen **20100807191947 Ignore-this: 13f306f7e9a768bd9772f7d7bf62ad05 ] [Oops, fix typos in previous patch Matthew Gruen **20100807180755 Ignore-this: 70961a76d963904b8db130ceebe7778f ] [Misc. minor improvements to the HTML interface Matthew Gruen **20100807180446 Ignore-this: b892819cc127714730c6bf267398a24 Add more links Double-checked authentication Search indices update when a package is uploaded ] [Small bugfixes for server setup Matthew Gruen **20100807101546 Ignore-this: 7bf38a70ae87dbae254f5e81ac34ef6b ] [Improve HTML generation Matthew Gruen **20100806152911 Ignore-this: 77c3fedc816236a1a48d3fd370023bd8 Touch up page aesthetics Update static files with some interim messages Add PackageList feature which keeps important package information handy for mass lists Expose very basic package-description text searching ] [Update Features and expose more HTML Matthew Gruen **20100804153711 Ignore-this: bbcdc91c4ce8a9bb6cd335d9742d636 Sorted reverse dependencies Calculated (user-immutable) tags Histograms go both item -> count and count -> item (descending) ] [Add filters for upload Matthew Gruen **20100804153124 Ignore-this: 1a6c923ee9d782c30b0b7617faab515c ] [Basic text searching feature, by package name and description, with crude OpenSearch plugin Matthew Gruen **20100804151112 Ignore-this: 31749ae318e46fc76f3492aa3ef9d557 ] [Changes to feature and underlying data structures Matthew Gruen **20100802153423 Ignore-this: 28bc82eeb4af3fe8ce748c320ab25910 Add NFData instances for Caches More information in a sort of centralized group index (which needs to be organized better) +Histogram datatype for cached sorted indices Enhance tag index, create tags from categories on initial import Serve documentation tarball ] [update authorspelling file Antoine Latter **20100803032450 Ignore-this: c83b27661ac33fa0389e62dd322a19a5 ] [Miscellaneous minor changes before running on sparky Matthew Gruen **20100730093022 Ignore-this: e34f71a75834abf74f0e38cb0ed55f79 ] [UserGroup development Matthew Gruen **20100730024439 Ignore-this: 5f9d0aff882426c0c7ce74587b23c757 Add central group listing Add HTML interfaces for user management and drop text Prepare to move registration to a different feature ] [Finish off preferred versions Matthew Gruen **20100730024328 Ignore-this: d1e42a38be4d9da81aca9d8aab218c7e Add HTML interface for it Implement atomic modifications of caches ] [Fix reverse dependencies for uploaded packages Matthew Gruen **20100727214518 Ignore-this: 88d79de1d5b758ad1f4f7e8b4ec50a0d ] [Development of Distribution.Server.Features.* modules and related modules Matthew Gruen **20100726154257 Ignore-this: 66122053a98f101268a6991ca8bc2170 ] [Update HTML generation code Matthew Gruen **20100726154022 Ignore-this: f0e658a75bc5503b76103067bcb48796 ] [Fixing import lists, miscellaneous adjustments Matthew Gruen **20100726153703 Ignore-this: 6d0340ffaf82dab095e34260eff9ae66 ] [Change ServerResponse type Matthew Gruen **20100706010245 Ignore-this: 7e81934da4c8ab8ac8f61c1b6fe629fe Get rid of Config argument Support for failing across formats ] [Improve cross-cutting interfaces Matthew Gruen **20100705164754 Ignore-this: 10f94f58926b012797980dd8a591dcb Add typed URI generation functions Add Documentation feature Allow multiple candidates ] [Revamp command line interface Matthew Gruen **20100702065855 Ignore-this: 43a3d1f44b6e876f9994aa34f6564111 Get import/export working ] [use deriveSerialize for complex data types Antoine Latter **20100627042226 Ignore-this: 6f7ddfa86a29b28e9b7c954cce7d07b9 ] [Implement candidate package support Matthew Gruen **20100625214332 Ignore-this: 8212dbae5ddbfeeb0aa6d00e42c28d91 In 200-something lines in Features/Check.hs with corresponding extension of Packages/{State,Types}.hs ] [Generalize Upload.hs interface Matthew Gruen **20100625214119 Ignore-this: 944d9652f2f6dd292031f3e77ab1c1ba Miscellaneous changes to cross-cutting code (auth type, fixing Modify* group update functions) Add DistroBackup.hs this time ] [Backup for build reports Matthew Gruen **20100624224421 Ignore-this: 2a976812b7534193dddf7aa85e500e9d ] [More revamping of import/export, adding DistroBackup.hs too Matthew Gruen **20100623004853 Ignore-this: 5847b7b3759315d1fbe4209d84d1263b ] [Make per-package build report ids and add reports feature Matthew Gruen **20100623004504 Ignore-this: 984ccb7d53c0c5cef63f1fed8e82a88 ] [Work on features, expanding distros and general backup interface Matthew Gruen **20100622145528 Ignore-this: c588f7e5c203a0a01bad3afef69f793c ] [Filled out PackageBackup.hs and continue decentralization of backup system Matthew Gruen **20100621170720 Ignore-this: 2ac364f20350868821ec20ee22ff3149 ] [Documentation for Resource.hs and improved URI generation interface Matthew Gruen **20100621170513 Ignore-this: 7de0d9a11be4f9855c3783d16cbb77bd ] [Implement mirror functionality (untested) and improve uploading Matthew Gruen **20100620011536 Ignore-this: 1a61b5398ed8c87f8133813849578151 ] [Update URI generation for Resources Matthew Gruen **20100620011259 Ignore-this: a41d4c72b73e0c77754235bd38b71d7b ] [Completed the newer complex routing system Matthew Gruen **20100617153243 Ignore-this: e4cba04b38a6b2bdf2aa160b6195ad06 ] [Add distro feature (Features.Distro) Matthew Gruen **20100616095534 Ignore-this: 49f726b209f22a900250e3c9641551e3 ] [Get hackage-server to compile under -Werror Matthew Gruen **20100616094933 Ignore-this: 8e1f7e0da6c0030ea2a5b48efc1ac918 ] [HTML feature with package page Matthew Gruen **20100615110643 Ignore-this: 71a844abef9dbb649f53bd3c6d9fae1e ] [Forgot to 'darcs add' some important modules Matthew Gruen **20100614191049 Ignore-this: e563153884be9d71044b980124435fff ] [Fill in upload feature Matthew Gruen **20100614043348 Ignore-this: f08fcfa561f00f042c55dd753f0c555f ] [Start to separate off package pages from the HTML representation thereof Matthew Gruen **20100613120755 Ignore-this: fa77001c8327d21e3cddb04a7949ea0e Move the ModuleForest into the Packages rather than Pages/Package dir. Dependency code still needs to be updated (undeprecated) and moved to the package page feature. ] [Minor: add some clarifying comments, remove crufty ones Matthew Gruen **20100613120502 Ignore-this: 79438d94e0c97c5710da317db6b911e2 ] [Working text/plain pages for the users feature Matthew Gruen **20100613061244 Ignore-this: ef1f213dddcfb14c709047aa1b0571fb ] [Module restructuring Matthew Gruen **20100612073042 Ignore-this: a1af495f80ffb55077fc23007c5269b9 Move backup-related items to Distribution.Server.Backup.* Add 8 fill-in-the-blank modules to Distribution.Server.Features.* ] [Add content types and Monoid instance to Resource; generalize ServerTree and Cache and improve ServerTree generation Matthew Gruen **20100612035515 Ignore-this: c55190457078a80d12970d0262c0781b ] [Reinvent UserGroup again and make auto-generated group Resources Matthew Gruen **20100612035307 Ignore-this: ad18d75b2a78cca7065f080babf3e1e8 ] [Switch to HackageFeature typeclass (and don't fragment top-level structures like Config as much) Matthew Gruen **20100612034453 Ignore-this: cc37c92c16a3f0c83872879b1032ef92 ] [Module imports, exports and movings related to some following changes Matthew Gruen **20100612033552 Ignore-this: 9d807d057d3ac3002bea25fba97c791d ] [The Accept header-parsing code I've been working on, from RFC 2616 14.1 Matthew Gruen **20100608094512 Ignore-this: 50fb4bfaf7d13bf302e4693279488d5 ] [Add guard for OPTIONS. Without it, the server returned blank responses when it should have 404'd Matthew Gruen **20100608094237 Ignore-this: be9fd0c7ab558ef543191e9a5d54e431 ] [Set up HackageAdmins and HackageTrustees components, part of Core and PackageUpload respectively Matthew Gruen **20100608094104 Ignore-this: 376f13aa7d59f575ac26abccdb918265 ] [Move Users-exporting functionality out of Export.hs (now in UserBackup.hs) Matthew Gruen **20100608093927 Ignore-this: c3a9622ae96cb41a103cd9770584b6b0 ] [Implement RestoreBackup for the Users type and integrate it into the server Matthew Gruen **20100608093200 Ignore-this: 97aacb623156e1f48adbdd9b5c6715e4 ] [Changes to data structures Matthew Gruen **20100607011708 Ignore-this: 7187829db91a2028c421160b75957009 Change data structures which happstack-state modifies to reflect internal structure. It compiles, while not guaranteed to all work. Notably, there are some regressions in the web interface, as HTML generation mostly hasn't been updated. Some infrastructure challenges remaining: modularizing the import/export system, adding flexible hooks, and shuffling around code for a more organized module tree ] [Implement Resource abstraction, integrate it into Feature, and integrate Feature into the main Server module Matthew Gruen **20100602171632 Ignore-this: cb0853a8db0a75303cbf0b3be89b660d ] [Revert changes to the library setup, since I've found a work-around Matthew Gruen **20100530033050 Ignore-this: 2b9f4bc8f6ff7105070ba0eb33875891 The library setup breaks on fresh installs, but can bootstrap itself on the executable setup. Revert to the version that works on fresh installs ] [Make the modules part of a library (not just executable) for easier testing wikigracenotes@gmail.com**20100530020001 Ignore-this: cf23660d4d86a8ab43ddb5de19dd905d ] [Switch to Happstack 0.5.* wikigracenotes@gmail.com**20100530015916 Ignore-this: 8d8b109e126357e24080c4352dd635b ] [Add Duncan's Feature modules, start moving towards something like that system wikigracenotes@gmail.com**20100530015641 Ignore-this: 1df6254aa7012162718472b30a6abdbd ] [Add digest function hackageDigestAuth implementing RFC 2617 wikigracenotes@gmail.com**20100530015529 Ignore-this: 7350f055bc2aee210d27bf59da12be79 ] [Add Show instances to many Hackage data structures wikigracenotes@gmail.com**20100530015254 Ignore-this: 624403665ccba410907f562fc773d81a ] [add authorspelling file Antoine Latter **20100404021327 Ignore-this: d169d51c9d3f562092e210d8b7005788 ] [make legacy support less ugly, include cgi-bin paths Antoine Latter **20100404020757 Ignore-this: c6c698dcb69aaeaef98014f00e42940c ] [-Wall clean Antoine Latter **20100207070536 Ignore-this: 4d3570414bc269ffa619be458878ceb ] [update TODO Antoine Latter **20100118055654 Ignore-this: f022493870a6a51573a0429c52f4514d ] [switch over to Duncan's work serving documentation directly from tarballs Antoine Latter **20100118053155 Ignore-this: 7298f9863dabb9091fc5e67e5c1432ca also cleanup of the .cabal file ] [improved legacy support Antoine Latter **20100102065912 Ignore-this: 6eaf33131354d09af22069fdc9bb8966 cabal update works cabal install works added documentation redirection added .cabal file redirection ] [turn on ability to display documentation links Antoine Latter **20091226213127 Ignore-this: 1469f33a3aea84472d5c502b6f165fd7 ] [Fixes to documentation upload Antoine Latter **20091226002113 Ignore-this: 1517aa208471a4eac3c9a52ce84a689 * Distribution.Server.Util.Serve.serveTarball works again. This isn't something we should ship, but it's good enough for testing. * Fix for uploading documentation to a package name Now, when we POST to /package/packageName/documentation it counts as a POST to /package/packageName-latestVersion/documentation The correpsonding GET has ben modified in the same fashion * When we GET a file in the documentation under the dirrectory /package/package-id/documentation we assume that all of the files in the tarball are under the directory (packageName packageId). We require this normalization as the same documentation can be seen from multiple URIs. ] [clean up build warnings Antoine Latter **20091225161237 Ignore-this: ff26c75554a6d784cce563d6b3ec7c95 ] [development done for distributions tracking is -Wall cleaner Antoine Latter **20091225052658 Ignore-this: a0fdd02aa362c8432fa82f1ca1a6af99 ] [import support for distributions tracking Antoine Latter **20091225050901 Ignore-this: 8d08a582f3a6a5019faec0193f292a4e ] [export distro information Antoine Latter **20091225040139 Ignore-this: e20a6875eedb2c03412be29e968929c3 Also: + fix merge error with export feature + fix some comments + restrict punctuation allowed in distro names ] [clean up permissions when deleting a distro Antoine Latter **20091224022940 Ignore-this: cc0871979d2bfda41cd39f4aa211afe7 ] [distro names are now first-class in the internal data structures Antoine Latter **20091224022022 Ignore-this: 9c0d760d64e8fab3d87926e04d029990 ] [distirbution tracking Antoine Latter **20091223205744 Ignore-this: 81b579e1a55a6d2e85244102eab6802a We now have a database to hold known distribtions which provide information about haskell packages they distribute. We have forms and put-actions to: - Create distributions - Allow specific users to manage the packages contained therin - Speficy which packages and at which versions a a particular distribution carries This informaiton now shows up in a packages page. ] [Update to happstack 0.4 Duncan Coutts **20091220033139 Ignore-this: e3e365a4bf85f0fc1a68dc82bc4c96f5 ] [Fix import module for PackageIndex Duncan Coutts **20091220030714 Ignore-this: 15ea970f0e3fac8c68cbe12378429d62 ] [Resolve conflicts in .cabal file Duncan Coutts **20091220030143 Ignore-this: 2be8c7745d44471e50485e439b1b1e79 ] [stop sending content length HTTP header for export tarball Antoine Latter **20091027013241 Ignore-this: a3435fc252ff810372edd0cc89fe30c WARNING: happstack likes to keep HTTP connections alive when we're not sending a content-length header. I haven't run into trouble with web-browsers, but if you're testing with wget you should call it like so: wget --header="Connection: close" ... ] [fix scaling issues in import/export aslatter@gmail.com**20091025043535 Ignore-this: a1a623d8fcbbe5125635f565e1425d15 import: added strictness export: added laziness ] [-Wall clean(er) aslatter@gmail.com**20091024194033 Ignore-this: 13ad9d6d0b215dd19ce9b52bc9b5539b There are a few warnings left, but I prefer the warned code the way it is. ] [use the same user name parser when adding users as when import users from the tarball aslatter@gmail.com**20091024181624 Ignore-this: a8b9dbd131bee19df32b7646d562c1bf ] [enumerating over all users now includes user ids in the reuslt aslatter@gmail.com**20091024181434 Ignore-this: b0abaafb5e229074d65df628f3bfba82 This was originally done in enumerateAllWithIds, but the enumeration is useless without including the ids, so it is now standard. ] [documentation cleanup aslatter@gmail.com**20091024181337 Ignore-this: 95f00f8a3e2c638dad72b5ed9dee524a ] [fixes for export/import aslatter@gmail.com**20091021033538 Ignore-this: 41de13aff8975589abe1745763ab056f - drop Data.Text for UTF8 wrangling - fix export of user ids ] [Import from exported tarball aslatter@gmail.com**20091021023414 Ignore-this: c66b828cfff31185e9fb89c5124797c5 ] [changes to other files to support import aslatter@gmail.com**20091021023310 Ignore-this: 438f3e9267ef2614f56e652de1db3da6 ] [updates to export to make streamier processing on import possible aslatter@gmail.com**20091021023131 Ignore-this: a44b0cef714adfed204d19b481316d2f ] [additions to Users and BuildReports for use in import aslatter@gmail.com**20091011031250 Ignore-this: 52e26d982cbe50499739b06e2890a228 ] [export server state to a tarball aslatter@gmail.com**20090914013940 Ignore-this: bc5e1aecb2cb2d95f157cdfdb1375e54 A 'GET' request on the URL 'admin/export.tar.gz' will dump the current state of the server to a tarball. This request requires admin rights. The export includes: - All versions of all packages - Users, user authentication details, user permissions - Uploaded buildreports (untested) - Uploaded documentation (untested) ] [update TODO aslatter@gmail.com**20091026033828 Ignore-this: c1c655dcbd39c20d664ece5e432f6a1b ] [changes for GHC 6.12 Antoine Latter **20091126191155 Ignore-this: bb0d5e32db9af061c65a471bbfa038c + Allow higher version of 'unix' package + Use newer version of 'time' package + Switch to Cabal-1.8.* - This requires defining our own version of the PackageIndex type The new versions of 'unix' and 'Cabal' are not yet on hackage. ] [More updates to package path URLs Antoine Latter **20091213034935 Ignore-this: 43598d9e665a92dfa925ae9880c018e9 ] [package path tweaks aslatter@gmail.com**20090909021849 Ignore-this: a7ccaeff58a6cc4341669139ce0390c4 ] [haddock fix aslatter@gmail.com**20090906171634 Ignore-this: d0eb49964facf1523a4a36294616dc6 haddock doesn't like these constructor field comments, so they are now de-haddockified. ] [Move some URLs about a bit Duncan Coutts **20090830142405] [Use consistent spelling of initialise in command line UI Duncan Coutts **20090829012117 Ignore-this: 872906d99fdccc7b5d05ac9ce8bde26e ] [Improve the error about missing static files dir Duncan Coutts **20090829012057 Ignore-this: 8268b9fc26b66da85b3e324f67452aa7 ] [Switch to port 8080 as the default Duncan Coutts **20090829011934 Ignore-this: 2f0bc505e396a3c89b407978d6bf72f6 Port 8080 is an official alternative http port. Port 5000 is already assigned for something else. ] [Revert a couple hlint suggestions Duncan Coutts **20090829011511 Ignore-this: f56c28abfafbbffb0b024fee4f869bf0 1. Using "unless condition" instead of "when (not condition)" is fine except that in the case that the condition is unlikely or an error it reads better to use when rather than unless. The word unless reads like we're expecting to run the code block except in some unlikely corner case, like "unless (null thingy) $ use thingy. 2. The [Char] vs String type alias. Yes, usually String makes sense but in this case our "string" is exactly 2 characters long. It is the characters that we are focusing on. We consider it as a pair of characters, not as some piece of text. ] [update TODO file aslatter@gmail.com**20090815202608 Ignore-this: 1af21e56c08c1869e31e5dc9091f88a ] [use 'updateCache' helper funtion consistently Antoine Latter **20090724044708 Ignore-this: 3f70b32c02ae4accda2ec448fbfac693 ] [hlint tweaks Antoine Latter **20090724044005 Ignore-this: 3ae39baad9dc19fb57bdee6bd5ee02c hlint wants to do more eta-reducing than I'm comfortable with. Other hlint suggestions have been ignored for style concerns. ] [URLs should only ever be POSIX style paths Antoine Latter **20090724041150 Ignore-this: b1a85bc80deff9394774d8ef661a8bbc ] [Simplify redundant code Duncan Coutts **20090724233542 The first arg of fileServe is the list of index files. So the admin.html is not appropriate for that. ] [Redo the top level exception handling, for better formatted output Duncan Coutts **20090724232509] [Move the importing / initialise code out of line Duncan Coutts **20090724232329] [Add server init sanity checking Duncan Coutts **20090724231821 Make sure people do not accidentally obliterate existing server state by doing an import or (re)init. Also check that if there is no existing server state that the user uses import or initialise to get things going, otherwise we'd end up with no admin user etc. Still todo: allow overriding these checks, and double check that when we do import or initialise, that we do get an admin user account. ] [Add some docs to the Server module Duncan Coutts **20090724231534] [Export Server function for checking if there is existing saved state Duncan Coutts **20090724230728 So we will be able to check people are not accidentally re-initialising the server state, or starting a server with no state at all. ] [Add a Unix signal hander for USR1 to write out a state checkpint Duncan Coutts **20090724033303 Usage: kill -USR1 $server_pid ] [Fix code indenting Duncan Coutts **20090724033224] [Shutdown the server properly Duncan Coutts **20090724033030] [Export functions to shutdown and checkpoint the server Duncan Coutts **20090724032746] [Switch to base 4 Duncan Coutts **20090721111452 rather than base 3 + extensible-exceptions ] [Remove redundant imports Duncan Coutts **20090721111156 As a result of splitting the state and server modules. ] [Split the Server and State modules into bits Duncan Coutts **20090721105342 Initially just packages and users. Some things still need moving about. ] [Move the Distribution.Server.Types module under Packages Duncan Coutts **20090721102031 It only contains the PkgInfo type these days ] [add package maintainer admin page Antoine Latter **20090720020648 Ignore-this: 1f3624a8bd82551e1652a65b89974193 * added a link on each package page to a package admin page * create package admin page which - Lists maintainers for a package - Allows adding maintainers - Allows removing maintainers * The admin page and actions are availble to: - anyone in the packagemaintainer group - anyone in the trustee group ] [Use package maintainer groups Antoine Latter **20090718030148 Ignore-this: 5823fac5abbfbd4f0b062aced8323385 * Add to pckage maintainer group when creating a new package * When doing a bulk import, every user who has uploaded a package goes into that package's maintainer group * Uploading a new version of an existing package requires memebership in the package maintainer group * Uploading documentation for a package requires membership in the package maintainer group ] [second try at bootstraping an admin user Antoine Latter **20090718014540 Ignore-this: df21b2cfe0232bf8962ebd55bb4dfbdc There are now two ways to do this: 1) As part of an import, the user can specify a file of newline-delimeted user names which should have admin privledges. A side effect of this feature is that a bulk-import not also resets permissions (which is important, since the UIDs in the permssions dd would be otherwise dangling) 2) A command line option to create a default admin user, with username "admin" and password "admin" This also clears out the same state as a bulk import. These options are mutually exclusive. ] [resolve conflicts/reconcile changes with tom's change password work Antoine Latter **20090715015311 Ignore-this: a6472161534c71492ab0f5b6e1b79c3b ] [user administration Antoine Latter **20090621201523 Ignore-this: 69155e10222f437a30405be09a67aeb7 Added primitives to Server.State for working with users Added a static html page for UI Added server-part as a back-end for the static html ] [Add a 'change password' feature thomas.dubuisson@gmail.com**20090708213226 Ignore-this: be61e3d17ae8f373f8556fbb4a8e5b6f ] [Upgrade to happstack 0.3.* thomas.dubuisson@gmail.com**20090708215452 Ignore-this: 75ecdac4154b5bf642d184bae266bbc0 ] [Alternative fix for finding crypt on linux, osx and solaris Duncan Coutts **20090704141444] [crypt.h doesn't exist for OS X, using unistd.h for OS X instead Antoine Latter **20090620145522 Ignore-this: a2e4689e58c10e59541abfbe92accd60 ] [Specify the correct version of the 'time' package thomas.dubuisson@gmail.com**20090608133635 Ignore-this: 9e80b527dc985ac48bcd6dfcd23095f7 ] [Serve packages with the proper name instead of 'tarball' thomas.dubuisson@gmail.com**20090608133553 Ignore-this: fe1792fc05a53fb4d8e26c57768b4882 ] [Add basic package checking capability. thomas.dubuisson@gmail.com**20090608131257 Ignore-this: 5c62cf7577c3a3a378a45096a87f2f9e ] [make it build under base3 or base4, and fix remaining warnings Simon Michael **20090605153133 Ignore-this: 57e0a7b1d2e4828a01ecc38406d95445 All but that one. ] [re-enabled commented out handlers Simon Michael **20090605145020 Ignore-this: 2be4dd3688170aaffeeb35fc8fe7f868 ] [a hasty port to Happstack 0.2 Simon Michael **20090605135345 Ignore-this: 64ca071809aea664a7e9d77000c8b85a Some server parts I couldn't figure have been commented out. ] [Switch to using the new tar package Duncan Coutts **20090301164933 And adjust to the API ] [Add TODO Duncan Coutts **20090221180522] [Add Distribution.Server.Util.Serve to other-modules field Duncan Coutts **20090221164535] [Fix topHandler Duncan Coutts **20090221155642] [Update to Happstack Duncan Coutts **20090221135026 Much easier than I expected! ] [add bug tracker link (#415) Duncan Coutts **20081127091430 merge of patch from Ross Paterson ] [Fix a bunch of warnings Duncan Coutts **20081126233126] [Sync changes in the package page from hackage-scripts Duncan Coutts **20081126231321 Add the module doc link changes. Update the haddock parser from 0.8 to 0.9. ] [Use base-3 and base-3 exceptions Duncan Coutts **20081126230648 For the moment. We can switch to base 4 later. ] [Bump the required version of HAppS from 0.9.2 to 0.9.3 Duncan Coutts **20081126181117] [Add support for documentation and (currently not used) groups. Lemmih **20081123003918 Ignore-this: 74c66a3636cbc08bb70c586b9d1cb2a1 ] [Run the web-server in its own thread. Lemmih **20081123003851 Ignore-this: 50f702047c7babab817734f8873e5c00 ] [Add tarball serving module. Lemmih **20081123003825 Ignore-this: 62efbb56a431e1831becc39d4ef1313f ] [HAppS-0.9.3 wibble. Lemmih **20081114214334] [GHC-6.10 exceptions. Lemmih **20081114214208] [Urk, forgot about ghc-6.10 exceptions. Lemmih **20081110160325] [Bump Cabal dependency to >=1.6.0 Lemmih **20081110003127] [Upgrade to Cabal-1.6.x Lemmih **20081110002830] [Add link to build logs on build report detail page Duncan Coutts **20080925070556] [Link build reports from package pages Duncan Coutts **20080925062110] [Update html for build reports, nicer presentation Duncan Coutts **20080925060726] [Fix BlobStorage for old as well as new bytestring Duncan Coutts **20080922033257 Subtle stuff with closing file handles ] [Add links to package deps Duncan Coutts **20080922020149 The feature was already there, just get the data to the right place. ] [Clarify some error conditions Duncan Coutts **20080918155440 Thanks to Tom DuBuisson ] [Fix BlogStorage's use of Handles Andrea Vezzosi **20080917193224] [Sync changes in static html pages Duncan Coutts **20080914203943] [Sync with Ross's recent changes in the package page Duncan Coutts **20080914193213] [Use new basic auth system Duncan Coutts **20080913233108] [Allow the auth group to be optional Duncan Coutts **20080913225449] [update basic auth server part to use new user infrastructure Duncan Coutts **20080913224426] [Fix importing and merging of accounts Duncan Coutts **20080913223612] [Switch the PkgInfo user to a UserId rather than String Duncan Coutts **20080913171821 Which means things that use it need the Users db if they want to convert that to a user name. ] [Move PkgInfo Binary instance into the same module as PkgInfo Duncan Coutts **20080913171406] [Use the common UserName type in the upload log entries Duncan Coutts **20080913162358] [Move shared code into another util module Duncan Coutts **20080913160748] [Split the index read/write functions into two modules Duncan Coutts **20080913145904] [Add users utils for converting user name <-> id Duncan Coutts **20080913145544] [Add users to the server state Duncan Coutts **20080913133505] [Add Binary instances for the User types Duncan Coutts **20080913133212] [Add command line args for importing accounts Duncan Coutts **20080913121146 and call the bulk import code. The information is not yet retained. ] [First part of support for importing user accounts Duncan Coutts **20080913121044] [Rearrange the bulk import slightly Duncan Coutts **20080913010514 So if it fails it should fail earlier ] [Simplify HtPasswdDb Duncan Coutts **20080913010209 It's just for parsing the files, not the in-memory structure ] [Document what UploadLog.read is returning Duncan Coutts **20080913005830] [Re-export Auth.Types from Users.Types Duncan Coutts **20080913005801] [Add Users.empty Duncan Coutts **20080913005730] [Add initial impl of users and groups Duncan Coutts **20080909024327 No ServerPart yet ] [Move the Upload module Duncan Coutts **20080908223838 In preperation to split out the other parts of upload from the State and Server modules. ] [Move some instances into their own module Duncan Coutts **20080907234556 Binary and typeable instances for Cabal types ] [Move some modules around Duncan Coutts **20080907234351] [Add initial support for standard htpasswd files Duncan Coutts **20080907182056 Not yet used. ] [Update to the latest tar code Duncan Coutts **20080907180715] [Get the static files from the cabal data-dir Duncan Coutts **20080907015524 With a command line override. Should mean it works 'out of the box' when built and installed via Cabal. ] [Rearrange server initialisation code Duncan Coutts **20080907014005 Better organsiation and abstraction layering ] [Rearrange server state dirs Duncan Coutts **20080906193008 Now put it all under one state dir which can be specified on the command line with --state-dir ] [Fix up list of data-files Duncan Coutts **20080906192859] [Cosmetic code changes in BlobStorage Duncan Coutts **20080906192214 Decided to leave the file layout as is, with the blobs in the top level dir rather than having two subdirs, one for blobs and the other for incomming. This dir will be a subdir of the server state dir anyway so an extra level is unnecessary. ] [Add an upper bound on the rss version Duncan Coutts **20080906191645 This is not strictly necessary but it helps the cabal-install constraint solver because later versions require a conflicting version of the HaXml package. ] [We do not need the process package anymore Duncan Coutts **20080906191609 Previously was used for calling tar and we now do that internally. ] [Use the complete list of other modules, or sdist does not work Duncan Coutts **20080906001319] [Consider only the base name of the uploaded file Duncan Coutts **20080903230520 Some clients send the whole path, including IE apparently and also existing versions of cabal-install ] [Fix up link to cabal-tiny.png Duncan Coutts **20080904141546] [Use new BuildReport parser Duncan Coutts **20080807222559] [Don't claim that Ross supports this rss feed. Duncan Coutts **20080807022906 Especially since we've not asked him! ] [Add the hackage favicon.ico Duncan Coutts **20080807021408] [Use XHtml strict with mime type application/xhtml+xml Duncan Coutts **20080807020411 and don't use http-equiv meta tags to specify the content-type. This is only for the generated pages. The static pages need fixing. ] [Add initial build report summary table Duncan Coutts **20080807003648] [Add initial per-package build report summary page Duncan Coutts **20080806234958] [Add PUT of build log Duncan Coutts **20080806225227] [Add GET of build report build log Duncan Coutts **20080806215242] [Add GET for /buildreports/$reportid Duncan Coutts **20080806210001] [Add initial POST for /buildreports Duncan Coutts **20080806205555] [Add instance Text BuildReportId Duncan Coutts **20080806205149] [Add event wrappers around the pure buildreports code. Lemmih **20080806205148] [Make parseBuildReport return Either not ParseResult Duncan Coutts **20080806204554] [Add "buildreports" top level uri Duncan Coutts **20080806202428] [Add BuildReports into the server state Duncan Coutts **20080806194954 added Binary instances etc ] [Add initial reporting code, internal representations Duncan Coutts **20080806124353] [Split Main into Main and Distribution.Server Duncan Coutts **20080805203935] [Very simple uploading. Lemmih **20080805002925] [Return the raw .cabal file from unpackPackage too Duncan Coutts **20080805000344] [Rewrite top level upload code, to enter the package into the store Duncan Coutts **20080804235157 Still does not add it to the index. ] [Add BlobStorage.addWith Duncan Coutts **20080804235016 Writes the input to the incomming area, gives us a chance to validate and then either rolls back or commits the blob to the store. Refactored internals to support both add and addWith. ] [Fix TarCheck test program Duncan Coutts **20080804172415] [Test code for Distribution.Server.Upload.unpackPackage Duncan Coutts **20080804122208] [Nearly working package upload Duncan Coutts **20080804021820 Checks the package tarball is ok. Does not yet check if it's in the index already or write it to the store or add it into the index. ] [Oops, the header name is Content-MD5 not MD5-Content Duncan Coutts **20080803135034] [Add ETag and Last-modified header to tarballs Duncan Coutts **20080803131954] [Add BlobId to PackageTarball resource type and add MD5-Content header Duncan Coutts **20080803105703 Split Tarball resource into PackageTarball and IndexTarball since they will use different headers, and we don't have a BlobId for the index tarball. ] [Fix warnings Duncan Coutts **20080802042950] [Move resource types and ToMessage instances to another module Duncan Coutts **20080803011306 These types will grow more meta data like times md5s etc and their instances will get more complex, to add corresponding http headers so it's sensible to move them out of Main now. ] [Add --import-archive flag and use it in main Duncan Coutts **20080801192851] [Open the BlobStorage in main and pass it inwards Duncan Coutts **20080801192815] [First go at merging tarballs in the BulkImport Duncan Coutts **20080801190629] [Support the URL scheme from the previous version of hackage. Lemmih **20080801182932] [Redirect /packages/00-index.tar.gz to just /00-index.tar.gz Duncan Coutts **20080801180625] [Move the 00-index.tar.gz under packages Duncan Coutts *-20080801180009 This is needed for cabal-install compatability ] [Move the 00-index.tar.gz under packages Duncan Coutts **20080801180009 This is needed for cabal-install compatability ] [Add tarball download and link from package pages Duncan Coutts **20080801175129] [Wibbles to the previous package. Lemmih **20080801175024] [Add pkgTarball :: Maybe BlobId to PkgInfo Duncan Coutts **20080801174553 And make BlobId an instance of Serialize ] [Add --host to override the hostname the server reports itself as Duncan Coutts **20080731005438 Needed if the service dns name is different from the machine name otherwise absolute self-links (eg in rss feed) will be wrong. For example community.haskell.org is served by nun.haskell.org ] [Get the server's host name on startup and use it in the rss feed Duncan Coutts **20080731004451 We need to provide absolute url links in the rss. ] [Put the username and group in the tarball index, because we can Duncan Coutts **20080731001348 Use the uploader as user name and "HackageDB" as the group name. ] [Fix up location of recent uploads page and rss feed Duncan Coutts **20080731000758] [Cache and output an RSS feed. Lemmih **20080730220538] [Limit rss feed to last 20 uploads Duncan Coutts **20080730215129] [Add rss feed generation, but not linked to a url yet Duncan Coutts **20080730213837] [Add recent page, not yet linked to a url Duncan Coutts **20080730194043] [Add caching for recent changes and the RSS feed. Lemmih **20080730193555] [Show the upload user in the package pages Duncan Coutts **20080729014541] [Add the new PkgInfo fields to the serialisation Duncan Coutts **20080729014503] [Include the upload user and old uploads in the PkgInfo Duncan Coutts **20080729013525 Report dropped log entries on import. ] [Use simple local die and log Duncan Coutts **20080729013231 We want a version of die that doesn't change spacing, since wewant to print log entries which are sensitive to white space. ] [Improve bulk import Duncan Coutts **20080728213355] [Move the bulk import code into a separate module Duncan Coutts **20080726063058 Currently trivial but we need to merge the index and log files into a single package index. ] [Add --port= command line option Duncan Coutts **20080726054050] [Rearrange option handling and initialisation Duncan Coutts **20080726053957 So we handle opts before initialising ] [Add GetOpt command line Duncan Coutts **20080726051508] [Add the download date to the package pages Duncan Coutts **20080725203201] [Add upload time for each package Duncan Coutts **20080725203015 Currently it gets the info from the filestamp in the tar index. We should check it with the upload log file. ] [Make the Index read/write specific to the PkgInfo type Duncan Coutts **20080725202204] [Fix up links on the front page Duncan Coutts **20080725145555] [Fix link to package description in package pages Duncan Coutts **20080725145524] [Clean up warnings Duncan Coutts **20080725144633] [Reimplement the cache to not use HAppS events Duncan Coutts **20080725142949 I'm still not sure it's perfect. ] [Don't rely on the Applicative instances in HAppS Duncan Coutts **20080725141422 So we can use the released version for the moment. ] [Minor wibble. Lemmih **20080724204929] [Simplify get of cabal file for versioned or unversioned packages Duncan Coutts **20080724214429] [Add extremely simple authentication. Lemmih **20080724203215] [Serve the .cabal file for each package version Duncan Coutts **20080724211630] [Doh! I can't spell Duncan Coutts **20080723195411] [Use absolute urls for package links Duncan Coutts **20080723195140] [Use 'method' instead of 'anyRequest'. Lemmih **20080723184557] [Fix links between package pages Duncan Coutts **20080723194036] [Wibbles. Lemmih **20080723183841] [Pass the other packages to the package page Duncan Coutts **20080723191748 and link up the unversioned page to the latest version ] [Simplify to just one state query Duncan Coutts **20080723191236] [Add none-versioned package page. Lemmih **20080723180629] [Add lookupPackageName state query Duncan Coutts **20080723185449] [Rename PackagesState to just State Duncan Coutts **20080723185007 It'll morph into the general server state. Obviously the state has packages in it but it's not the whole thing. ] [Fix loads of warnings Duncan Coutts **20080723034459 The remaining warning are valid indications of things we've not finished properly yet. ] [Read .cabal files in the state as UTF-8 Duncan Coutts **20080723034208 The .cabal files really are UTF-8 format. So we must decode them before parsing. This is important to be able to display people's names correctly. The XHtml code escapes high unicode chars because the overall content type is iso-8859-1 ] [Fix the links in the index to individual packages Duncan Coutts **20080723031843 though those urls don't actually exist yet. ] [Fix the url in the package search Duncan Coutts **20080723031821] [Add status message to indicate when it's done starting up Duncan Coutts **20080723022806] [Add built with cabal icon Duncan Coutts **20080723022732] [Use "get" rather than "GET" in form method Duncan Coutts **20080723022004 The W3C validator complains about "GET". ] [Fix up the links in the static html pages Duncan Coutts **20080723021928] [First go at integrating the package pages Duncan Coutts **20080723012309] [Fix a couple of urls. Lemmih **20080722203331] [Overwrite the old cache instead of updating it. Lemmih **20080722203303] [Change caching strategy to an asynchronous model. Lemmih **20080722183629] [Add first part of package pages. Duncan Coutts **20080722150210 Code to construct the PackageData is not in yet. ] [Serve package index page at /packages/ Duncan Coutts **20080722145546] [Comment out non-compiling code in Unpack module Duncan Coutts **20080722145401] [Hack the Unpack module a bit more Duncan Coutts **20080721235950 Needs to move towards being completely pure. ] [Add accounts.html static page Duncan Coutts **20080721234029] [First go at integrating the package index page Duncan Coutts **20080721233749] [Update .cabal file Duncan Coutts **20080721215257] [Serv hackage.html by default. Lemmih **20080721184516] [Fix link to upload.html Lemmih **20080721183226] [Add upload.html Duncan Coutts **20080721193052] [Add some of the static html pages from the current hackage Duncan Coutts **20080721192602 and serve them up ] [Fix caching bug. Lemmih **20080721180818] [Support uploading of the 00-index.tar.gz file. Lemmih **20080721180751] [Combine the unpack/upload code from hackage-scripts into one module Duncan Coutts **20080721144807 And eliminate and simplify in the process. ] [Move modules from Hackage/ to Distribution/Server Duncan Coutts **20080721144739] [Rewrite tar handling again Duncan Coutts **20080721124702 It can now round-trip archives in v7, ustar and gnu formats. And it's still not done. The error handling is no good. ] [Compile with -Wall Duncan Coutts **20080718114848] [Fill in the impl of generatePackageIndex Duncan Coutts **20080718113928 generating the tarball from the package index, that we cache in the in-memory server state ] [Use the new tarball code Duncan Coutts **20080718113906] [Trim the code we stole from hackage-scripts Duncan Coutts **20080718113657 and make it compile ] [Tidy and simplify Hackage.Types Duncan Coutts **20080718113620] [Further improvements to the tar code Duncan Coutts **20080718113547] [Rewrite the tar code and reading/writing the package index Duncan Coutts **20080717222634] [Add non-functioning download support. Lemmih **20080717212007] [Support simple uploading. Lemmih **20080717210040] [Steal tar utilities from hackage-scripts. Lemmih **20080717210007] [Minor changes to the BlobStore. Lemmih **20080717205837] [Use HAppS-State as the database. Lemmih **20080717175626] [Crude hacks to make it compile (with Cabal 1.4) Eelco Lempsink **20080629124225] [Flesh out the blob storage stuff a bit Duncan Coutts **20080624180319] [Result of hacking in the pub with Lemmih Duncan Coutts **20080624110314] [Initial hello-world style demo prog. Duncan Coutts **20080622192441] Patch bundle hash: b8b9b4969233b6b0b1aecec74b84033ad3d6e50d